Hi, Stephan,
I see your concern and I totally agree with you.
The interface on RM side is now `Map<String key, String/Long value>
getYarn/KubernetesExternalResource()`. The only valid information RM
get from it is the configuration key of that external resource in
Yarn/K8s. The "String/Long value" would be the same as the
external-resource.{resourceName}.amount.
So, I think it makes sense to replace these two interfaces with two
configs, i.e. external-resource.{resourceName}.yarn/kubernetes.key. We
may lose some extensibility, but AFAIK it could work with common
external resources like GPU, FPGA. WDYT?
Best,
Yangze Guo
On Fri, Mar 27, 2020 at 7:59 PM Stephan Ewen <se...@apache.org> wrote:
>
> Maybe one final comment: It is probably not an issue, but let's try and
> keep user code (via user code classloader) out of the ResourceManager, if
> possible.
>
> As background:
>
> There were thoughts in the past to support setups where the RM must run
> with "superuser" credentials, but we cannot run JM/TM with these
> credentials, as the user code might access them otherwise.
> This is actually possible today, you can run the RM in a different JVM or
> in a different container, and give it more credentials than JMs / TMs. But
> for this to be feasible, we cannot allow any user-defined code to be in the
> JVM, because that instantaneously breaks the isolation of credentials.
>
>
>
> On Fri, Mar 27, 2020 at 4:01 AM Yangze Guo <karma...@gmail.com> wrote:
>
> > Thanks for the feedback, @Till and @Xintong.
> >
> > Regarding separating the interface, I'm also +1 with it.
> >
> > Regarding the resource allocation interface, true, it's dangerous to
> > give much access to user codes. Changing the return type to Map<String
> > key, String/Long value> makes sense to me. AFAIK, it is compatible
> > with all the first-party supported resources for Yarn/Kubernetes. It
> > could also free us from the potential dependency issue as well.
> >
> > Best,
> > Yangze Guo
> >
> > On Fri, Mar 27, 2020 at 10:42 AM Xintong Song <tonysong...@gmail.com>
> > wrote:
> > >
> > > Thanks for updating the FLIP, Yangze.
> > >
> > > I agree with Till that we probably want to separate the K8s/Yarn
> > decorator
> > > calls. Users can still configure one driver class, and we can use
> > > `instanceof` to check whether the driver implemented K8s/Yarn specific
> > > interfaces.
> > >
> > > Moreover, I'm not sure about exposing entire `ContainerRequest` / `Pod`
> > > (`AbstractKubernetesStepDecorator` directly manipulates on `Pod`) to user
> > > codes. It gives more access to user codes than needed for defining
> > external
> > > resource, which might cause problems. Instead, I would suggest to have
> > > interface like `Map<String key, String value>
> > > getYarn/KubernetesExternalResource()` and assemble them into
> > > `ContainerRequest` / `Pod` in Yarn/KubernetesResourceManager.
> > >
> > > Thank you~
> > >
> > > Xintong Song
> > >
> > >
> > >
> > > On Fri, Mar 27, 2020 at 1:10 AM Till Rohrmann <trohrm...@apache.org>
> > wrote:
> > >
> > > > Hi everyone,
> > > >
> > > > I'm a bit late to the party. I think the current proposal looks good.
> > > >
> > > > Concerning the ExternalResourceDriver interface defined in the FLIP
> > [1], I
> > > > would suggest to not include the decorator calls for Kubernetes and
> > Yarn in
> > > > the base interface. Instead I would suggest to segregate the deployment
> > > > specific decorator calls into separate interfaces. That way an
> > > > ExternalResourceDriver does not have to support all deployments from
> > the
> > > > very beginning. Moreover, some resources might not be supported by a
> > > > specific deployment target and the natural way to express this would
> > be to
> > > > not implement the respective deployment specific interface.
> > > >
> > > > Moreover, having void
> > > > addExternalResourceToRequest(AMRMClient.ContainerRequest
> > containerRequest)
> > > > in the ExternalResourceDriver interface would require Hadoop on Flink's
> > > > classpath whenever the external resource driver is being used.
> > > >
> > > > [1]
> > > >
> > > >
> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-108%3A+Add+GPU+support+in+Flink
> > > >
> > > > Cheers,
> > > > Till
> > > >
> > > > On Thu, Mar 26, 2020 at 12:45 PM Stephan Ewen <se...@apache.org>
> > wrote:
> > > >
> > > > > Nice, thanks a lot!
> > > > >
> > > > > On Thu, Mar 26, 2020 at 10:21 AM Yangze Guo <karma...@gmail.com>
> > wrote:
> > > > >
> > > > > > Thanks for the suggestion, @Stephan, @Becket and @Xintong.
> > > > > >
> > > > > > I've updated the FLIP accordingly. I do not add a
> > > > > > ResourceInfoProvider. Instead, I introduce the
> > ExternalResourceDriver,
> > > > > > which takes the responsibility of all relevant operations on both
> > RM
> > > > > > and TM sides.
> > > > > > After a rethink about decoupling the management of external
> > resources
> > > > > > from TaskExecutor, I think we could do the same thing on the
> > > > > > ResourceManager side. We do not need to add a specific allocation
> > > > > > logic to the ResourceManager each time we add a specific external
> > > > > > resource.
> > > > > > - For Yarn, we need the ExternalResourceDriver to edit the
> > > > > > containerRequest.
> > > > > > - For Kubenetes, ExternalResourceDriver could provide a decorator
> > for
> > > > > > the TM pod.
> > > > > >
> > > > > > In this way, just like MetricReporter, we allow users to define
> > their
> > > > > > custom ExternalResourceDriver. It is more extensible and fits the
> > > > > > separation of concerns. For more details, please take a look at
> > [1].
> > > > > >
> > > > > > [1]
> > > > > >
> > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-108%3A+Add+GPU+support+in+Flink
> > > > > >
> > > > > > Best,
> > > > > > Yangze Guo
> > > > > >
> > > > > > On Wed, Mar 25, 2020 at 7:32 PM Stephan Ewen <se...@apache.org>
> > wrote:
> > > > > > >
> > > > > > > This sounds good to go ahead from my side.
> > > > > > >
> > > > > > > I like the approach that Becket suggested - in that case the core
> > > > > > > abstraction that everyone would need to understand would be
> > "external
> > > > > > > resource allocation" and the "ResourceInfoProvider", and the GPU
> > > > > specific
> > > > > > > code would be a specific implementation only known to that
> > component
> > > > > that
> > > > > > > allocates the external resource. That fits the separation of
> > concerns
> > > > > > well.
> > > > > > >
> > > > > > > I also understand that it should not be over-engineered in the
> > first
> > > > > > > version, so some simplification makes sense, and then gradually
> > > > expand
> > > > > > from
> > > > > > > there.
> > > > > > >
> > > > > > > So +1 to go ahead with what was suggested above (Xintong /
> > Becket)
> > > > from
> > > > > > my
> > > > > > > side.
> > > > > > >
> > > > > > > On Mon, Mar 23, 2020 at 6:55 AM Xintong Song <
> > tonysong...@gmail.com>
> > > > > > wrote:
> > > > > > >
> > > > > > > > Thanks for the comments, Stephan & Becket.
> > > > > > > >
> > > > > > > > @Stephan
> > > > > > > >
> > > > > > > > I see your concern, and I completely agree with you that we
> > should
> > > > > > first
> > > > > > > > think about the "library" / "plugin" / "extension" style if
> > > > possible.
> > > > > > > >
> > > > > > > > If GPUs are sliced and assigned during scheduling, there may be
> > > > > reason,
> > > > > > > > > although it looks that it would belong to the slot then. Is
> > that
> > > > > > what we
> > > > > > > > > are doing here?
> > > > > > > >
> > > > > > > >
> > > > > > > > In the current proposal, we do not have the GPUs sliced and
> > > > assigned
> > > > > to
> > > > > > > > slots, because it could be problematic without dynamic slot
> > > > > allocation.
> > > > > > > > E.g., the number of GPUs might not be evenly divisible by the
> > > > number
> > > > > of
> > > > > > > > slots.
> > > > > > > >
> > > > > > > > I think it makes sense to eventually have the GPUs assigned to
> > > > slots.
> > > > > > Even
> > > > > > > > then, we might still need a TM level GPUManager (or
> > > > ResourceProvider
> > > > > > like
> > > > > > > > Becket suggested). For memory, in each slot we can simply
> > request
> > > > the
> > > > > > > > amount of memory, leaving it to JVM / OS to decide which memory
> > > > > > (address)
> > > > > > > > should be assigned. For GPU, and potentially other resources
> > like
> > > > > > FPGA, we
> > > > > > > > need to explicitly specify which GPU (index) should be used.
> > > > > > Therefore, we
> > > > > > > > need some component at the TM level to coordinate which slot
> > uses
> > > > > which
> > > > > > > > GPU.
> > > > > > > >
> > > > > > > > IMO, unless we say Flink will not support slot-level GPU
> > slicing at
> > > > > > least
> > > > > > > > in the foreseeable future, I don't see a good way to avoid
> > touching
> > > > > > the TM
> > > > > > > > core. To that end, I think Becket's suggestion points to a good
> > > > > > direction,
> > > > > > > > that supports more features (GPU, FPGA, etc.) with less
> > coupling to
> > > > > > the TM
> > > > > > > > core (only needs to understand the general interfaces). The
> > > > detailed
> > > > > > > > implementation for specific resource types can even be
> > encapsulated
> > > > > as
> > > > > > a
> > > > > > > > library.
> > > > > > > >
> > > > > > > > @Becket
> > > > > > > >
> > > > > > > > Thanks for sharing your thought on the final state. Despite the
> > > > > > details how
> > > > > > > > the interfaces should look like, I think this is a really good
> > > > > > abstraction
> > > > > > > > for supporting general resource types.
> > > > > > > >
> > > > > > > > I'd like to further clarify that, the following three things
> > are
> > > > all
> > > > > > that
> > > > > > > > the "Flink core" needs to understand.
> > > > > > > >
> > > > > > > > - The *amount* of resource, for scheduling. Actually, we
> > already
> > > > > > have
> > > > > > > > the Resource class in ResourceProfile and ResourceSpec for
> > > > > extended
> > > > > > > > resource. It's just not really used.
> > > > > > > > - The *info*, that Flink provides to the operators / user
> > codes.
> > > > > > > > - The *provider*, which generates the info based on the
> > amount.
> > > > > > > >
> > > > > > > > The "core" does not need to understand the specific
> > implementation
> > > > > > details
> > > > > > > > of the above three. They can even be implemented in a 3rd-party
> > > > > > library.
> > > > > > > > Similar to how we allow users to define their custom
> > > > MetricReporter.
> > > > > > > >
> > > > > > > > Thank you~
> > > > > > > >
> > > > > > > > Xintong Song
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Mon, Mar 23, 2020 at 8:45 AM Becket Qin <
> > becket....@gmail.com>
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > Thanks for the comment, Stephan.
> > > > > > > > >
> > > > > > > > > - If everything becomes a "core feature", it will make the
> > > > > project
> > > > > > hard
> > > > > > > > > > to develop in the future. Thinking "library" / "plugin" /
> > > > > > "extension"
> > > > > > > > > style
> > > > > > > > > > where possible helps.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Completely agree. It is much more important to design a
> > mechanism
> > > > > > than
> > > > > > > > > focusing on a specific case. Here is what I am thinking to
> > fully
> > > > > > support
> > > > > > > > > custom resource management:
> > > > > > > > > 1. On the JM / RM side, use ResourceProfile and ResourceSpec
> > to
> > > > > > define
> > > > > > > > the
> > > > > > > > > resource and the amount required. They will be used to find
> > > > > suitable
> > > > > > TMs
> > > > > > > > > slots to run the tasks. At this point, the resources are only
> > > > > > measured by
> > > > > > > > > amount, i.e. they do not have individual ID.
> > > > > > > > >
> > > > > > > > > 2. On the TM side, have something like
> > *"ResourceInfoProvider"*
> > > > to
> > > > > > > > identify
> > > > > > > > > and provides the detail information of the individual
> > resource,
> > > > > e.g.
> > > > > > GPU
> > > > > > > > > ID.. It is important because the operator may have to
> > explicitly
> > > > > > interact
> > > > > > > > > with the physical resource it uses. The ResourceInfoProvider
> > > > might
> > > > > > look
> > > > > > > > > like something below.
> > > > > > > > > interface ResourceInfoProvider<INFO> {
> > > > > > > > > Map<AbstractID, INFO> retrieveResourceInfo(OperatorId
> > opId,
> > > > > > > > > ResourceProfile resourceProfile);
> > > > > > > > > }
> > > > > > > > >
> > > > > > > > > - There could be several "*ResourceInfoProvider*" configured
> > on
> > > > the
> > > > > > TM to
> > > > > > > > > retrieve the information for different resources.
> > > > > > > > > - The TM will be responsible to assign those individual
> > resources
> > > > > to
> > > > > > each
> > > > > > > > > operator according to their requested amount.
> > > > > > > > > - The operators will be able to get the ResourceInfo from
> > their
> > > > > > > > > RuntimeContext.
> > > > > > > > >
> > > > > > > > > If we agree this is a reasonable final state. We can adapt
> > the
> > > > > > current
> > > > > > > > FLIP
> > > > > > > > > to it. In fact it does not sound a big change to me. All the
> > > > > proposed
> > > > > > > > > configuration can be as is, it is just that Flink itself
> > won't
> > > > care
> > > > > > about
> > > > > > > > > them, instead a GPUInfoProviver implementing the
> > > > > ResourceInfoProvider
> > > > > > > > will
> > > > > > > > > use them.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > >
> > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > >
> > > > > > > > > On Mon, Mar 23, 2020 at 1:47 AM Stephan Ewen <
> > se...@apache.org>
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi all!
> > > > > > > > > >
> > > > > > > > > > The main point I wanted to throw into the discussion is the
> > > > > > following:
> > > > > > > > > > - With more and more use cases, more and more tools go
> > into
> > > > > Flink
> > > > > > > > > > - If everything becomes a "core feature", it will make
> > the
> > > > > > project
> > > > > > > > hard
> > > > > > > > > > to develop in the future. Thinking "library" / "plugin" /
> > > > > > "extension"
> > > > > > > > > style
> > > > > > > > > > where possible helps.
> > > > > > > > > >
> > > > > > > > > > - A good thought experiment is always: How many future
> > > > > developers
> > > > > > > > have
> > > > > > > > > to
> > > > > > > > > > interact with this code (and possibly understand it
> > partially),
> > > > > > even if
> > > > > > > > > the
> > > > > > > > > > features they touch have nothing to do with GPU support. If
> > > > many
> > > > > > > > > > contributors to unrelated features will have to touch it
> > and
> > > > > > understand
> > > > > > > > > it,
> > > > > > > > > > then let's think if there is a different solution. Maybe
> > there
> > > > is
> > > > > > not,
> > > > > > > > > but
> > > > > > > > > > then we should be sure why.
> > > > > > > > > >
> > > > > > > > > > - That led me to raising this issue: If the GPU manager
> > > > > becomes a
> > > > > > > > core
> > > > > > > > > > service in the TaskManager, Environment, RuntimeContext,
> > etc.
> > > > > then
> > > > > > > > > everyone
> > > > > > > > > > developing TM and streaming tasks need to understand the
> > GPU
> > > > > > manager.
> > > > > > > > > That
> > > > > > > > > > seems oddly specific, is my impression.
> > > > > > > > > >
> > > > > > > > > > Access to configuration seems not the right reason to do
> > that.
> > > > We
> > > > > > > > should
> > > > > > > > > > expose the Flink configuration from the RuntimeContext
> > anyways.
> > > > > > > > > >
> > > > > > > > > > If GPUs are sliced and assigned during scheduling, there
> > may be
> > > > > > reason,
> > > > > > > > > > although it looks that it would belong to the slot then. Is
> > > > that
> > > > > > what
> > > > > > > > we
> > > > > > > > > > are doing here?
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Stephan
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Fri, Mar 20, 2020 at 2:58 AM Xintong Song <
> > > > > > tonysong...@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Thanks for the feedback, Becket.
> > > > > > > > > > >
> > > > > > > > > > > IMO, eventually an operator should only see info of GPUs
> > that
> > > > > are
> > > > > > > > > > dedicated
> > > > > > > > > > > for it, instead of all GPUs on the machine/container in
> > the
> > > > > > current
> > > > > > > > > > design.
> > > > > > > > > > > It does not make sense to let the user who writes a UDF
> > to
> > > > > worry
> > > > > > > > about
> > > > > > > > > > > coordination among multiple operators running on the same
> > > > > > machine.
> > > > > > > > And
> > > > > > > > > if
> > > > > > > > > > > we want to limit the GPU info an operator sees, we
> > should not
> > > > > > let the
> > > > > > > > > > > operator to instantiate GPUManager, which means we have
> > to
> > > > > expose
> > > > > > > > > > something
> > > > > > > > > > > through runtime context, either GPU info or some kind of
> > > > > limited
> > > > > > > > access
> > > > > > > > > > to
> > > > > > > > > > > the GPUManager.
> > > > > > > > > > >
> > > > > > > > > > > Thank you~
> > > > > > > > > > >
> > > > > > > > > > > Xintong Song
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Thu, Mar 19, 2020 at 5:48 PM Becket Qin <
> > > > > becket....@gmail.com
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > It probably make sense for us to first agree on the
> > final
> > > > > > state.
> > > > > > > > More
> > > > > > > > > > > > specifically, will the resource info be exposed through
> > > > > runtime
> > > > > > > > > context
> > > > > > > > > > > > eventually?
> > > > > > > > > > > >
> > > > > > > > > > > > If that is the final state and we have a seamless
> > migration
> > > > > > story
> > > > > > > > > from
> > > > > > > > > > > this
> > > > > > > > > > > > FLIP to that final state, Personally I think it is OK
> > to
> > > > > > expose the
> > > > > > > > > GPU
> > > > > > > > > > > > info in the runtime context.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks,
> > > > > > > > > > > >
> > > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon, Mar 16, 2020 at 11:21 AM Xintong Song <
> > > > > > > > tonysong...@gmail.com
> > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > @Yangze,
> > > > > > > > > > > > > I think what Stephan means (@Stephan, please correct
> > me
> > > > if
> > > > > > I'm
> > > > > > > > > wrong)
> > > > > > > > > > > is
> > > > > > > > > > > > > that, we might not need to hold and maintain the
> > > > GPUManager
> > > > > > as a
> > > > > > > > > > > service
> > > > > > > > > > > > in
> > > > > > > > > > > > > TaskManagerServices or RuntimeContext. An
> > alternative is
> > > > to
> > > > > > > > create
> > > > > > > > > /
> > > > > > > > > > > > > retrieve the GPUManager only in the operators that
> > need
> > > > it,
> > > > > > e.g.,
> > > > > > > > > > with
> > > > > > > > > > > a
> > > > > > > > > > > > > static method `GPUManager.get()`.
> > > > > > > > > > > > >
> > > > > > > > > > > > > @Stephan,
> > > > > > > > > > > > > I agree with you on excluding GPUManager from
> > > > > > > > TaskManagerServices.
> > > > > > > > > > > > >
> > > > > > > > > > > > > - For the first step, where we provide unified
> > > > TM-level
> > > > > > GPU
> > > > > > > > > > > > information
> > > > > > > > > > > > > to all operators, it should be fine to have
> > operators
> > > > > > access /
> > > > > > > > > > > > > lazy-initiate GPUManager by themselves.
> > > > > > > > > > > > > - In future, we might have some more fine-grained
> > GPU
> > > > > > > > > management,
> > > > > > > > > > > > where
> > > > > > > > > > > > > we need to maintain GPUManager as a service and
> > put
> > > > GPU
> > > > > > info
> > > > > > > > in
> > > > > > > > > > slot
> > > > > > > > > > > > > profiles. But at least for now it's not necessary
> > to
> > > > > > introduce
> > > > > > > > > > such
> > > > > > > > > > > > > complexity.
> > > > > > > > > > > > >
> > > > > > > > > > > > > However, I have some concerns on excluding GPUManager
> > > > from
> > > > > > > > > > > RuntimeContext
> > > > > > > > > > > > > and let operators access it directly.
> > > > > > > > > > > > >
> > > > > > > > > > > > > - Configurations needed for creating the
> > GPUManager is
> > > > > not
> > > > > > > > > always
> > > > > > > > > > > > > available for operators.
> > > > > > > > > > > > > - If later we want to have fine-grained control
> > over
> > > > GPU
> > > > > > > > (e.g.,
> > > > > > > > > > > > > operators in each slot can only see GPUs reserved
> > for
> > > > > that
> > > > > > > > > slot),
> > > > > > > > > > > the
> > > > > > > > > > > > > approach cannot be easily extended.
> > > > > > > > > > > > >
> > > > > > > > > > > > > I would suggest to wrap the GPUManager behind
> > > > > RuntimeContext
> > > > > > and
> > > > > > > > > only
> > > > > > > > > > > > > expose the GPUInfo to users. For now, we can declare
> > a
> > > > > method
> > > > > > > > > > > > > `getGPUInfo()` in RuntimeContext, with a default
> > > > definition
> > > > > > that
> > > > > > > > > > calls
> > > > > > > > > > > > > `GPUManager.get()` to get the lazily-created
> > GPUManager.
> > > > If
> > > > > > later
> > > > > > > > > we
> > > > > > > > > > > want
> > > > > > > > > > > > > to create / retrieve GPUManager in a different way,
> > we
> > > > can
> > > > > > simply
> > > > > > > > > > > change
> > > > > > > > > > > > > how `getGPUInfo` is implemented, without needing to
> > > > change
> > > > > > any
> > > > > > > > > public
> > > > > > > > > > > > > interfaces.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thank you~
> > > > > > > > > > > > >
> > > > > > > > > > > > > Xintong Song
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Sat, Mar 14, 2020 at 10:09 AM Yangze Guo <
> > > > > > karma...@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > @Shephan
> > > > > > > > > > > > > > Do you mean Minicluster? Yes, it makes sense to
> > share
> > > > the
> > > > > > GPU
> > > > > > > > > > Manager
> > > > > > > > > > > > > > in such scenario.
> > > > > > > > > > > > > > If that's what you worry about, I'm +1 for holding
> > > > > > > > > > > > > > GPUManager(ExternalResourceManagers) in
> > TaskExecutor
> > > > > > instead of
> > > > > > > > > > > > > > TaskManagerServices.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Regarding the RuntimeContext/FunctionContext, it
> > just
> > > > > > holds the
> > > > > > > > > GPU
> > > > > > > > > > > > > > info instead of the GPU Manager. AFAIK, it's the
> > only
> > > > > > place we
> > > > > > > > > > could
> > > > > > > > > > > > > > pass GPU info to the
> > RichFunction/UserDefinedFunction.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > Yangze Guo
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Sat, Mar 14, 2020 at 4:06 AM Isaac Godfried <
> > > > > > > > > > is...@paddlesoft.net
> > > > > > > > > > > >
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > ---- On Fri, 13 Mar 2020 15:58:20 +0000
> > > > > se...@apache.org
> > > > > > > > wrote
> > > > > > > > > > > ----
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Can we somehow keep this out of the
> > TaskManager
> > > > > > services
> > > > > > > > > > > > > > > > I fear that we could not. IMO, the
> > GPUManager(or
> > > > > > > > > > > > > > > > ExternalServicesManagers in future) is
> > conceptually
> > > > > > one of
> > > > > > > > > the
> > > > > > > > > > > task
> > > > > > > > > > > > > > > > manager services, just like MemoryManager
> > before
> > > > > 1.10.
> > > > > > > > > > > > > > > > - It maintains/holds the GPU resource at TM
> > level
> > > > and
> > > > > > all
> > > > > > > > of
> > > > > > > > > > the
> > > > > > > > > > > > > > > > operators allocate the GPU resources from it.
> > So,
> > > > it
> > > > > > should
> > > > > > > > > be
> > > > > > > > > > > > > > > > exclusive to a single TaskExecutor.
> > > > > > > > > > > > > > > > - We could add a collection called
> > > > > > ExternalResourceManagers
> > > > > > > > > to
> > > > > > > > > > > hold
> > > > > > > > > > > > > > > > all managers of other external resources in the
> > > > > future.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Can you help me understand why this needs the
> > > > addition
> > > > > in
> > > > > > > > > > > > > > TaskMagerServices
> > > > > > > > > > > > > > > or in the RuntimeContext?
> > > > > > > > > > > > > > > Are you worried about the case when multiple Task
> > > > > > Executors
> > > > > > > > run
> > > > > > > > > > in
> > > > > > > > > > > > the
> > > > > > > > > > > > > > same
> > > > > > > > > > > > > > > JVM? That's not common, but wouldn't it actually
> > be
> > > > > good
> > > > > > in
> > > > > > > > > that
> > > > > > > > > > > case
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > share the GPU Manager, given that the GPU is
> > shared?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > Stephan
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > ---------------------------
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > What parts need information about this?
> > > > > > > > > > > > > > > > In this FLIP, operators need the information.
> > Thus,
> > > > > we
> > > > > > > > expose
> > > > > > > > > > GPU
> > > > > > > > > > > > > > > > information to the
> > RuntimeContext/FunctionContext.
> > > > > The
> > > > > > slot
> > > > > > > > > > > profile
> > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > not aware of GPU resources as GPU is TM level
> > > > > resource
> > > > > > now.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Can the GPU Manager be a "self contained"
> > thing
> > > > > that
> > > > > > > > simply
> > > > > > > > > > > takes
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > configuration, and then abstracts everything
> > > > > > internally?
> > > > > > > > > > > > > > > > Yes, we just pass the path/args of the discover
> > > > > script
> > > > > > and
> > > > > > > > > how
> > > > > > > > > > > many
> > > > > > > > > > > > > > > > GPUs per TM to it. It takes the responsibility
> > to
> > > > get
> > > > > > the
> > > > > > > > GPU
> > > > > > > > > > > > > > > > information and expose them to the
> > > > > > > > > > RuntimeContext/FunctionContext
> > > > > > > > > > > > of
> > > > > > > > > > > > > > > > Operators. Meanwhile, we'd better not allow
> > > > operators
> > > > > > to
> > > > > > > > > > directly
> > > > > > > > > > > > > > > > access GPUManager, it should get what they want
> > > > from
> > > > > > > > Context.
> > > > > > > > > > We
> > > > > > > > > > > > > could
> > > > > > > > > > > > > > > > then decouple the interface/implementation of
> > > > > > GPUManager
> > > > > > > > and
> > > > > > > > > > > Public
> > > > > > > > > > > > > > > > API.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > Yangze Guo
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Fri, Mar 13, 2020 at 7:26 PM Stephan Ewen <
> > > > > > > > > se...@apache.org
> > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > It sounds fine to initially start with GPU
> > > > specific
> > > > > > > > support
> > > > > > > > > > and
> > > > > > > > > > > > > think
> > > > > > > > > > > > > > > > about
> > > > > > > > > > > > > > > > > generalizing this once we better understand
> > the
> > > > > > space.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > About the implementation suggested in
> > FLIP-108:
> > > > > > > > > > > > > > > > > - Can we somehow keep this out of the
> > TaskManager
> > > > > > > > services?
> > > > > > > > > > > > > Anything
> > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > have to pull through all layers of the TM
> > makes
> > > > the
> > > > > > TM
> > > > > > > > > > > components
> > > > > > > > > > > > > yet
> > > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > complex and harder to maintain.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > - What parts need information about this?
> > > > > > > > > > > > > > > > > -> do the slot profiles need information
> > about
> > > > the
> > > > > > GPU?
> > > > > > > > > > > > > > > > > -> Can the GPU Manager be a "self contained"
> > > > thing
> > > > > > that
> > > > > > > > > > simply
> > > > > > > > > > > > > takes
> > > > > > > > > > > > > > > > > the configuration, and then abstracts
> > everything
> > > > > > > > > internally?
> > > > > > > > > > > > > > Operators
> > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > access it via "GPUManager.get()" or so?
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Wed, Mar 4, 2020 at 4:19 AM Yangze Guo <
> > > > > > > > > > karma...@gmail.com>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thanks for all the feedbacks.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > @Becket
> > > > > > > > > > > > > > > > > > Regarding the WebUI and GPUInfo, you're
> > right,
> > > > > > I'll add
> > > > > > > > > > them
> > > > > > > > > > > to
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > Public API section.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > @Stephan @Becket
> > > > > > > > > > > > > > > > > > Regarding the general extended resource
> > > > > mechanism,
> > > > > > I
> > > > > > > > > second
> > > > > > > > > > > > > > Xintong's
> > > > > > > > > > > > > > > > > > suggestion.
> > > > > > > > > > > > > > > > > > - It's better to leverage ResourceProfile
> > and
> > > > > > > > > ResourceSpec
> > > > > > > > > > > > after
> > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > supporting fine-grained GPU scheduling. As
> > a
> > > > > first
> > > > > > step
> > > > > > > > > > > > > proposal, I
> > > > > > > > > > > > > > > > > > prefer to not include it in the scope of
> > this
> > > > > FLIP.
> > > > > > > > > > > > > > > > > > - Regarding the "Extended Resource
> > Manager",
> > > > if I
> > > > > > > > > > understand
> > > > > > > > > > > > > > > > > > correctly, it just a code refactoring atm,
> > we
> > > > > could
> > > > > > > > > extract
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > open/close/allocateExtendResources of
> > > > GPUManager
> > > > > to
> > > > > > > > that
> > > > > > > > > > > > > > interface. If
> > > > > > > > > > > > > > > > > > that is the case, +1 to do it during
> > > > > > implementation.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > @Xingbo
> > > > > > > > > > > > > > > > > > As Xintong said, we looked into how Spark
> > > > > supports
> > > > > > a
> > > > > > > > > > general
> > > > > > > > > > > > > > "Custom
> > > > > > > > > > > > > > > > > > Resource Scheduling" before and decided to
> > > > > > introduce a
> > > > > > > > > > common
> > > > > > > > > > > > > > resource
> > > > > > > > > > > > > > > > > > configuration
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > >
> > schema(taskmanager.resource.{resourceName}.amount/discovery-script)
> > > > > > > > > > > > > > > > > > to make it more extensible. I think the
> > > > > "resource"
> > > > > > is a
> > > > > > > > > > > proper
> > > > > > > > > > > > > > level
> > > > > > > > > > > > > > > > > > to contain all the configs of extended
> > > > resources.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > > Yangze Guo
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Wed, Mar 4, 2020 at 10:48 AM Xingbo
> > Huang <
> > > > > > > > > > > > hxbks...@gmail.com
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Thanks a lot for the FLIP, Yangze.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > There is no doubt that GPU resource
> > > > management
> > > > > > > > support
> > > > > > > > > > will
> > > > > > > > > > > > > > greatly
> > > > > > > > > > > > > > > > > > > facilitate the development of AI-related
> > > > > > applications
> > > > > > > > > by
> > > > > > > > > > > > > PyFlink
> > > > > > > > > > > > > > > > users.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > I have only one comment about this wiki:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Regarding the names of several GPU
> > > > > > configurations, I
> > > > > > > > > > think
> > > > > > > > > > > it
> > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > better
> > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > delete the resource field makes it
> > consistent
> > > > > > with
> > > > > > > > the
> > > > > > > > > > > names
> > > > > > > > > > > > of
> > > > > > > > > > > > > > other
> > > > > > > > > > > > > > > > > > > resource-related configurations in
> > > > > > TaskManagerOption.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > e.g.
> > > > > > taskmanager.resource.gpu.discovery-script.path
> > > > > > > > ->
> > > > > > > > > > > > > > > > > > > taskmanager.gpu.discovery-script.path
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Xingbo
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Xintong Song <tonysong...@gmail.com>
> > > > > > 于2020年3月4日周三
> > > > > > > > > > > 上午10:39写道：
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > @Stephan, @Becket,
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Actually, Yangze, Yang and I also had
> > an
> > > > > > offline
> > > > > > > > > > > discussion
> > > > > > > > > > > > > > about
> > > > > > > > > > > > > > > > > > making
> > > > > > > > > > > > > > > > > > > > the "GPU Support" as some general
> > "Extended
> > > > > > > > Resource
> > > > > > > > > > > > > Support".
> > > > > > > > > > > > > > We
> > > > > > > > > > > > > > > > > > believe
> > > > > > > > > > > > > > > > > > > > supporting extended resources in a
> > general
> > > > > > > > mechanism
> > > > > > > > > is
> > > > > > > > > > > > > > definitely
> > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > good
> > > > > > > > > > > > > > > > > > > > and extensible way. The reason we
> > propose
> > > > > this
> > > > > > FLIP
> > > > > > > > > > > > narrowing
> > > > > > > > > > > > > > its
> > > > > > > > > > > > > > > > scope
> > > > > > > > > > > > > > > > > > > > down to GPU alone, is mainly for the
> > > > concern
> > > > > on
> > > > > > > > extra
> > > > > > > > > > > > efforts
> > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > review
> > > > > > > > > > > > > > > > > > > > capacity needed for a general
> > mechanism.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > To come up with a well design on a
> > general
> > > > > > extended
> > > > > > > > > > > > resource
> > > > > > > > > > > > > > > > management
> > > > > > > > > > > > > > > > > > > > mechanism, we would need to investigate
> > > > more
> > > > > > on how
> > > > > > > > > > > people
> > > > > > > > > > > > > use
> > > > > > > > > > > > > > > > > > different
> > > > > > > > > > > > > > > > > > > > kind of resources in practice. For
> > GPU, we
> > > > > > learnt
> > > > > > > > > such
> > > > > > > > > > > > > > knowledge
> > > > > > > > > > > > > > > > from
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > experts, Becket and his team members.
> > But
> > > > for
> > > > > > FPGA,
> > > > > > > > > or
> > > > > > > > > > > > other
> > > > > > > > > > > > > > > > potential
> > > > > > > > > > > > > > > > > > > > extended resources, we don't have such
> > > > > > convenient
> > > > > > > > > > > > information
> > > > > > > > > > > > > > > > sources,
> > > > > > > > > > > > > > > > > > > > making the investigation requires more
> > > > > efforts,
> > > > > > > > > which I
> > > > > > > > > > > > tend
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > think
> > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > not necessary atm.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > On the other hand, we also looked into
> > how
> > > > > > Spark
> > > > > > > > > > > supports a
> > > > > > > > > > > > > > general
> > > > > > > > > > > > > > > > > > "Custom
> > > > > > > > > > > > > > > > > > > > Resource Scheduling". Assuming we want
> > to
> > > > > have
> > > > > > a
> > > > > > > > > > similar
> > > > > > > > > > > > > > general
> > > > > > > > > > > > > > > > > > extended
> > > > > > > > > > > > > > > > > > > > resource mechanism in the future, we
> > > > believe
> > > > > > that
> > > > > > > > the
> > > > > > > > > > > > current
> > > > > > > > > > > > > > GPU
> > > > > > > > > > > > > > > > > > support
> > > > > > > > > > > > > > > > > > > > design can be easily extended, in an
> > > > > > incremental
> > > > > > > > way
> > > > > > > > > > > > without
> > > > > > > > > > > > > > too
> > > > > > > > > > > > > > > > many
> > > > > > > > > > > > > > > > > > > > reworks.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > - The most important part is probably
> > user
> > > > > > > > > interfaces.
> > > > > > > > > > > > Spark
> > > > > > > > > > > > > > > > offers
> > > > > > > > > > > > > > > > > > > > configuration options to define the
> > amount,
> > > > > > > > discovery
> > > > > > > > > > > > script
> > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > vendor
> > > > > > > > > > > > > > > > > > > > (on
> > > > > > > > > > > > > > > > > > > > k8s) in a per resource type bias [1],
> > which
> > > > > is
> > > > > > very
> > > > > > > > > > > similar
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > what
> > > > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > > proposed in this FLIP. I think it's not
> > > > > > necessary
> > > > > > > > to
> > > > > > > > > > > expose
> > > > > > > > > > > > > > > > config
> > > > > > > > > > > > > > > > > > > > options
> > > > > > > > > > > > > > > > > > > > in the general way atm, since we do not
> > > > have
> > > > > > > > supports
> > > > > > > > > > for
> > > > > > > > > > > > > other
> > > > > > > > > > > > > > > > > > resource
> > > > > > > > > > > > > > > > > > > > types now. If later we decided to have
> > per
> > > > > > resource
> > > > > > > > > > type
> > > > > > > > > > > > > config
> > > > > > > > > > > > > > > > > > > > options, we
> > > > > > > > > > > > > > > > > > > > can have backwards compatibility on the
> > > > > current
> > > > > > > > > > proposed
> > > > > > > > > > > > > > options
> > > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > simple key mapping.
> > > > > > > > > > > > > > > > > > > > - For the GPU Manager, if later needed
> > we
> > > > can
> > > > > > > > change
> > > > > > > > > it
> > > > > > > > > > > to
> > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > "Extended
> > > > > > > > > > > > > > > > > > > > Resource Manager" (or whatever it is
> > > > called).
> > > > > > That
> > > > > > > > > > should
> > > > > > > > > > > > be
> > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > pure
> > > > > > > > > > > > > > > > > > > > component-internal refactoring.
> > > > > > > > > > > > > > > > > > > > - For ResourceProfile and ResourceSpec,
> > > > there
> > > > > > are
> > > > > > > > > > already
> > > > > > > > > > > > > > > > fields for
> > > > > > > > > > > > > > > > > > > > general extended resource. We can of
> > course
> > > > > > > > leverage
> > > > > > > > > > them
> > > > > > > > > > > > > when
> > > > > > > > > > > > > > > > > > > > supporting
> > > > > > > > > > > > > > > > > > > > fine grained GPU scheduling. That is
> > also
> > > > not
> > > > > > in
> > > > > > > > the
> > > > > > > > > > > scope
> > > > > > > > > > > > of
> > > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > first
> > > > > > > > > > > > > > > > > > > > step proposal, and would require
> > FLIP-56 to
> > > > > be
> > > > > > > > > finished
> > > > > > > > > > > > > first.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > To summary up, I agree with Becket that
> > > > have
> > > > > a
> > > > > > > > > separate
> > > > > > > > > > > > FLIP
> > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > general extended resource mechanism,
> > and
> > > > keep
> > > > > > it in
> > > > > > > > > > mind
> > > > > > > > > > > > when
> > > > > > > > > > > > > > > > > > discussing
> > > > > > > > > > > > > > > > > > > > and implementing the current one.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Thank you~
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Xintong Song
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > [1]
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > https://spark.apache.org/docs/3.0.0-preview/configuration.html#custom-resource-scheduling-and-configuration-overview
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > On Wed, Mar 4, 2020 at 9:18 AM Becket
> > Qin <
> > > > > > > > > > > > > > becket....@gmail.com>
> > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > That's a good point, Stephan. It
> > makes
> > > > > total
> > > > > > > > sense
> > > > > > > > > to
> > > > > > > > > > > > > > generalize
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > resource management to support custom
> > > > > > resources.
> > > > > > > > > > Having
> > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > allows
> > > > > > > > > > > > > > > > > > users
> > > > > > > > > > > > > > > > > > > > > to add new resources by themselves.
> > The
> > > > > > general
> > > > > > > > > > > resource
> > > > > > > > > > > > > > > > management
> > > > > > > > > > > > > > > > > > may
> > > > > > > > > > > > > > > > > > > > > involve two different aspects:
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 1. The custom resource type
> > definition.
> > > > It
> > > > > is
> > > > > > > > > > supported
> > > > > > > > > > > > by
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > extended
> > > > > > > > > > > > > > > > > > > > > resources in ResourceProfile and
> > > > > > ResourceSpec.
> > > > > > > > This
> > > > > > > > > > > will
> > > > > > > > > > > > > > likely
> > > > > > > > > > > > > > > > cover
> > > > > > > > > > > > > > > > > > > > > majority of the cases.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 2. The custom resource allocation
> > logic,
> > > > > > i.e. how
> > > > > > > > > to
> > > > > > > > > > > > assign
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > resources
> > > > > > > > > > > > > > > > > > > > > to different tasks, operators, and
> > so on.
> > > > > > This
> > > > > > > > may
> > > > > > > > > > > > require
> > > > > > > > > > > > > > two
> > > > > > > > > > > > > > > > > > levels /
> > > > > > > > > > > > > > > > > > > > > steps:
> > > > > > > > > > > > > > > > > > > > > a. Subtask level - make sure the
> > subtasks
> > > > > > are put
> > > > > > > > > > into
> > > > > > > > > > > > > > > > suitable
> > > > > > > > > > > > > > > > > > > > slots.
> > > > > > > > > > > > > > > > > > > > > It is done by the global RM and is
> > not
> > > > > > > > customizable
> > > > > > > > > > > right
> > > > > > > > > > > > > > now.
> > > > > > > > > > > > > > > > > > > > > b. Operator level - map the exact
> > > > resource
> > > > > > to the
> > > > > > > > > > > > operators
> > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > TM.
> > > > > > > > > > > > > > > > > > > > e.g.
> > > > > > > > > > > > > > > > > > > > > GPU 1 for operator A, GPU 2 for
> > operator
> > > > B.
> > > > > > This
> > > > > > > > > step
> > > > > > > > > > > is
> > > > > > > > > > > > > > needed
> > > > > > > > > > > > > > > > > > assuming
> > > > > > > > > > > > > > > > > > > > > the global RM does not distinguish
> > > > > individual
> > > > > > > > > > resources
> > > > > > > > > > > > of
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > same
> > > > > > > > > > > > > > > > > > type.
> > > > > > > > > > > > > > > > > > > > > It is true for memory, but not for
> > GPU.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > The GPU manager is designed to do 2.b
> > > > here.
> > > > > > So it
> > > > > > > > > > > should
> > > > > > > > > > > > > > > > discover the
> > > > > > > > > > > > > > > > > > > > > physical GPU information and
> > bind/match
> > > > > them
> > > > > > to
> > > > > > > > > each
> > > > > > > > > > > > > > operators.
> > > > > > > > > > > > > > > > > > Making
> > > > > > > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > > > > general will fill in the missing
> > piece to
> > > > > > support
> > > > > > > > > > > custom
> > > > > > > > > > > > > > resource
> > > > > > > > > > > > > > > > > > type
> > > > > > > > > > > > > > > > > > > > > definition. But I'd avoid calling it
> > a
> > > > > > "External
> > > > > > > > > > > Resource
> > > > > > > > > > > > > > > > Manager" to
> > > > > > > > > > > > > > > > > > > > avoid
> > > > > > > > > > > > > > > > > > > > > confusion with RM, maybe something
> > like
> > > > > > "Operator
> > > > > > > > > > > > Resource
> > > > > > > > > > > > > > > > Assigner"
> > > > > > > > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > > > > > be more accurate. So for each
> > resource
> > > > type
> > > > > > users
> > > > > > > > > can
> > > > > > > > > > > > have
> > > > > > > > > > > > > an
> > > > > > > > > > > > > > > > > > optional
> > > > > > > > > > > > > > > > > > > > > "Operator Resource Assigner" in the
> > TM.
> > > > For
> > > > > > > > memory,
> > > > > > > > > > > users
> > > > > > > > > > > > > > don't
> > > > > > > > > > > > > > > > need
> > > > > > > > > > > > > > > > > > > > this,
> > > > > > > > > > > > > > > > > > > > > but for other extended resources,
> > users
> > > > may
> > > > > > need
> > > > > > > > > > that.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Personally I think a pluggable
> > "Operator
> > > > > > Resource
> > > > > > > > > > > > Assigner"
> > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > achievable
> > > > > > > > > > > > > > > > > > > > > in this FLIP. But I am also OK with
> > > > having
> > > > > > that
> > > > > > > > in
> > > > > > > > > a
> > > > > > > > > > > > > separate
> > > > > > > > > > > > > > > > FLIP
> > > > > > > > > > > > > > > > > > > > because
> > > > > > > > > > > > > > > > > > > > > the interface between the "Operator
> > > > > Resource
> > > > > > > > > > Assigner"
> > > > > > > > > > > > and
> > > > > > > > > > > > > > > > operator
> > > > > > > > > > > > > > > > > > may
> > > > > > > > > > > > > > > > > > > > > take a while to settle down if we
> > want to
> > > > > > make it
> > > > > > > > > > > > generic.
> > > > > > > > > > > > > > But I
> > > > > > > > > > > > > > > > > > think
> > > > > > > > > > > > > > > > > > > > our
> > > > > > > > > > > > > > > > > > > > > implementation should take this
> > future
> > > > work
> > > > > > into
> > > > > > > > > > > > > > consideration so
> > > > > > > > > > > > > > > > > > that we
> > > > > > > > > > > > > > > > > > > > > don't need to break backwards
> > > > compatibility
> > > > > > once
> > > > > > > > we
> > > > > > > > > > > have
> > > > > > > > > > > > > > that.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > On Wed, Mar 4, 2020 at 12:27 AM
> > Stephan
> > > > > Ewen
> > > > > > <
> > > > > > > > > > > > > > se...@apache.org>
> > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Thank you for writing this FLIP.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > I cannot really give much input
> > into
> > > > the
> > > > > > > > > mechanics
> > > > > > > > > > of
> > > > > > > > > > > > > > GPU-aware
> > > > > > > > > > > > > > > > > > > > > scheduling
> > > > > > > > > > > > > > > > > > > > > > and GPU allocation, as I have no
> > > > > experience
> > > > > > > > with
> > > > > > > > > > > that.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > One thought I had when reading the
> > > > > > proposal is
> > > > > > > > if
> > > > > > > > > > it
> > > > > > > > > > > > > makes
> > > > > > > > > > > > > > > > sense to
> > > > > > > > > > > > > > > > > > > > look
> > > > > > > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > > > > > > > > the "GPU Manager" as an "External
> > > > > Resource
> > > > > > > > > > Manager",
> > > > > > > > > > > > and
> > > > > > > > > > > > > > GPU
> > > > > > > > > > > > > > > > is one
> > > > > > > > > > > > > > > > > > > > such
> > > > > > > > > > > > > > > > > > > > > > resource.
> > > > > > > > > > > > > > > > > > > > > > The way I understand the
> > > > ResourceProfile
> > > > > > and
> > > > > > > > > > > > > ResourceSpec,
> > > > > > > > > > > > > > > > that is
> > > > > > > > > > > > > > > > > > how
> > > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > > is done there.
> > > > > > > > > > > > > > > > > > > > > > It has the advantage that it looks
> > more
> > > > > > > > > extensible.
> > > > > > > > > > > > Maybe
> > > > > > > > > > > > > > > > there is
> > > > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > > GPU
> > > > > > > > > > > > > > > > > > > > > > Resource, a specialized NVIDIA GPU
> > > > > > Resource,
> > > > > > > > and
> > > > > > > > > > FPGA
> > > > > > > > > > > > > > > > Resource, a
> > > > > > > > > > > > > > > > > > > > Alibaba
> > > > > > > > > > > > > > > > > > > > > > TPU Resource, etc.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > > > > > > Stephan
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > On Tue, Mar 3, 2020 at 7:57 AM
> > Becket
> > > > > Qin <
> > > > > > > > > > > > > > > > becket....@gmail.com>
> > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Thanks for the FLIP Yangze. GPU
> > > > > resource
> > > > > > > > > > management
> > > > > > > > > > > > > > support
> > > > > > > > > > > > > > > > is a
> > > > > > > > > > > > > > > > > > > > > > must-have
> > > > > > > > > > > > > > > > > > > > > > > for machine learning use cases.
> > > > > Actually
> > > > > > it
> > > > > > > > is
> > > > > > > > > > one
> > > > > > > > > > > of
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > mostly
> > > > > > > > > > > > > > > > > > > > asked
> > > > > > > > > > > > > > > > > > > > > > > question from the users who are
> > > > > > interested in
> > > > > > > > > > using
> > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > for ML.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Some quick comments / questions
> > to
> > > > the
> > > > > > wiki.
> > > > > > > > > > > > > > > > > > > > > > > 1. The WebUI / REST API should
> > > > probably
> > > > > > also
> > > > > > > > be
> > > > > > > > > > > > > > mentioned in
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > public
> > > > > > > > > > > > > > > > > > > > > > > interface section.
> > > > > > > > > > > > > > > > > > > > > > > 2. Is the data structure that
> > holds
> > > > GPU
> > > > > > info
> > > > > > > > > > also a
> > > > > > > > > > > > > > public
> > > > > > > > > > > > > > > > API?
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > On Tue, Mar 3, 2020 at 10:15 AM
> > > > Xintong
> > > > > > Song
> > > > > > > > <
> > > > > > > > > > > > > > > > > > tonysong...@gmail.com>
> > > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Thanks for drafting the FLIP
> > and
> > > > > > kicking
> > > > > > > > off
> > > > > > > > > > the
> > > > > > > > > > > > > > > > discussion,
> > > > > > > > > > > > > > > > > > > > Yangze.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Big +1 for this feature.
> > Supporting
> > > > > > using
> > > > > > > > of
> > > > > > > > > > GPU
> > > > > > > > > > > in
> > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > significant,
> > > > > > > > > > > > > > > > > > > > > > > > especially for the ML
> > scenarios.
> > > > > > > > > > > > > > > > > > > > > > > > I've reviewed the FLIP wiki
> > doc and
> > > > > it
> > > > > > > > looks
> > > > > > > > > > good
> > > > > > > > > > > > to
> > > > > > > > > > > > > > me. I
> > > > > > > > > > > > > > > > > > think
> > > > > > > > > > > > > > > > > > > > > it's a
> > > > > > > > > > > > > > > > > > > > > > > > very good first step for
> > Flink's
> > > > GPU
> > > > > > > > > supports.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Thank you~
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Xintong Song
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > On Mon, Mar 2, 2020 at 12:06 PM
> > > > > Yangze
> > > > > > Guo
> > > > > > > > <
> > > > > > > > > > > > > > > > karma...@gmail.com
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Hi everyone,
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > We would like to start a
> > > > discussion
> > > > > > > > thread
> > > > > > > > > on
> > > > > > > > > > > > > > "FLIP-108:
> > > > > > > > > > > > > > > > Add
> > > > > > > > > > > > > > > > > > GPU
> > > > > > > > > > > > > > > > > > > > > > > > > support in Flink"[1].
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > This FLIP mainly discusses
> > the
> > > > > > following
> > > > > > > > > > > issues:
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > - Enable user to configure
> > how
> > > > many
> > > > > > GPUs
> > > > > > > > > in a
> > > > > > > > > > > > task
> > > > > > > > > > > > > > > > executor
> > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > > > forward such requirements to
> > the
> > > > > > external
> > > > > > > > > > > > resource
> > > > > > > > > > > > > > > > managers
> > > > > > > > > > > > > > > > > > (for
> > > > > > > > > > > > > > > > > > > > > > > > > Kubernetes/Yarn/Mesos
> > setups).
> > > > > > > > > > > > > > > > > > > > > > > > > - Provide information of
> > > > available
> > > > > > GPU
> > > > > > > > > > > resources
> > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > operators.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Key changes proposed in the
> > FLIP
> > > > > are
> > > > > > as
> > > > > > > > > > > follows:
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > - Forward GPU resource
> > > > requirements
> > > > > > to
> > > > > > > > > > > > > > Yarn/Kubernetes.
> > > > > > > > > > > > > > > > > > > > > > > > > - Introduce GPUManager as
> > one of
> > > > > the
> > > > > > task
> > > > > > > > > > > manager
> > > > > > > > > > > > > > > > services to
> > > > > > > > > > > > > > > > > > > > > > discover
> > > > > > > > > > > > > > > > > > > > > > > > > and expose GPU resource
> > > > information
> > > > > > to
> > > > > > > > the
> > > > > > > > > > > > context
> > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > functions.
> > > > > > > > > > > > > > > > > > > > > > > > > - Introduce the default
> > script
> > > > for
> > > > > > GPU
> > > > > > > > > > > discovery,
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > which we
> > > > > > > > > > > > > > > > > > > > > provide
> > > > > > > > > > > > > > > > > > > > > > > > > the privilege mode to help
> > user
> > > > to
> > > > > > > > achieve
> > > > > > > > > > > > > > worker-level
> > > > > > > > > > > > > > > > > > isolation
> > > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > > > > standalone mode.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Please find more details in
> > the
> > > > > FLIP
> > > > > > wiki
> > > > > > > > > > > > document
> > > > > > > > > > > > > > [1].
> > > > > > > > > > > > > > > > > > Looking
> > > > > > > > > > > > > > > > > > > > > > forward
> > > > > > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > > > your feedbacks.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > [1]
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-108%3A+Add+GPU+support+in+Flink
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > > > > > > > > > Yangze Guo
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> >
