[
https://issues.apache.org/jira/browse/YARN-8851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16654557#comment-16654557
]
Zhankun Tang edited comment on YARN-8851 at 10/18/18 3:48 AM:
--------------------------------------------------------------
[~leftnoteasy]
5) DeviceRuntimeSpec is empty, what you plan to add?
{color:#d04437}Zhankun–>{color} It's a set of classeds and is described more
clearly in the UML figure of the design doc. In general, it is returned by
vendor plugin's implementation hook "OnDeviceAllocated" to describe the
requirements of environment or volume creation or docker command updates.etc.
This DeviceRuntimeSpec will be translated to YARN internal operations by the
"DevicePluginAdapter". For instance, GPUv1 might require a volume creation
before container launch. And in this DeviceRuntimeSpec, it is a volumeClaim to
describe it and let NM to create it. Another example is GPUv2 needs additional
environment when running Docker container, this is described by "envs". And for
cgroups device isolation, it is described by "MountDeviceSpec".
The class is like this:
{code:java}
class DeviceRuntimeSpec {
String runtime; // describe needed runtime. maybe not needed?
Map<String, String> envs; // describe needed environment variables before using
devices
Set<MountVolumeSpec> volumeMounts; // describe volumes need to be mounted
before using devices
Set<MountDeviceSpec> devices; // describe devices needed to be mount
Set<VolumeSpec> volumeClaim; // describe volume to be created/delete before
using devices
}
{code}
was (Author: tangzhankun):
[~leftnoteasy]
5) DeviceRuntimeSpec is empty, what you plan to add?
{color:#d04437}Zhankun–>{color} It's a set of classeds and is described more
clearly in the UML figure of the design doc. In general, it is returned by
vendor plugin's implementation hook "OnDeviceAllocated" to describe the
requirements of environment or volume creation or docker command updates.etc.
This DeviceRuntimeSpec will be translated to YARN internal operations by the
"DevicePluginAdapter". For instance, GPUv1 might require a volume creation
before container launch. And in this DeviceRuntimeSpec, it is a volumeClaim to
describe it and let NM to create it. Another example is GPUv2 needs additional
environment when running Docker container, this is described by "envs". And for
cgroups device isolation, it is described by "MountDeviceSpec".
The class is like this:
{code:java}
class DeviceRuntimeSpec {
Map<String, String> envs; // describe needed environment variables before using
devices
Set<MountVolumeSpec> volumeMounts; // describe volumes need to be mounted
before using devices
Set<MountDeviceSpec> devices; // describe devices needed to be mount
Set<VolumeSpec> volumeClaim; // describe volume to be created/delete before
using devices
}
{code}
> [Umbrella] A new pluggable device plugin framework to ease vendor plugin
> development
> ------------------------------------------------------------------------------------
>
> Key: YARN-8851
> URL: https://issues.apache.org/jira/browse/YARN-8851
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: yarn
> Reporter: Zhankun Tang
> Assignee: Zhankun Tang
> Priority: Major
> Attachments: YARN-8851-WIP2-trunk.001.patch,
> YARN-8851-WIP3-trunk.001.patch, YARN-8851-WIP4-trunk.001.patch, [YARN-8851]
> YARN_New_Device_Plugin_Framework_Design_Proposal-3.pdf, [YARN-8851]
> YARN_New_Device_Plugin_Framework_Design_Proposal.pdf
>
>
> At present, we support GPU/FPGA device in YARN through a native, coupling
> way. But it's difficult for a vendor to implement such a device plugin
> because the developer needs much knowledge of YARN internals. And this brings
> burden to the community to maintain both YARN core and vendor-specific code.
> Here we propose a new device plugin framework to ease vendor device plugin
> development and provide a more flexible way to integrate with YARN NM.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
