Re: [Qemu-devel] [RFC PATCH v2 1/3] vGPU Core driver

[Jike Song]

On 03/01/2016 07:17 AM, Neo Jia wrote:
> On Mon, Feb 29, 2016 at 05:39:02AM +, Tian, Kevin wrote:
>>> From: Kirti Wankhede
>>> Sent: Wednesday, February 24, 2016 12:24 AM
>>>
>>> Signed-off-by: Kirti Wankhede 
>>> Signed-off-by: Neo Jia 
>>
>> Hi, Kirti/Neo,
>>
>> Thanks a lot for you updated version. Having not looked into detail
>> code, first come with some high level comments.
>>
>> First, in a glimpse the majority of the code (possibly >95%) is device
>> agnostic, though we call it vgpu today. Just thinking about the
>> extensibility and usability of this framework, would it be better to 
>> name it in a way that any other type of I/O device can be fit into 
>> this framework? I don't have a good idea of the name now, but 
>> a simple idea is to replace vgpu with vdev (vdev-core, vfio-vdev,
>> vfio-iommu-type1-vdev, etc.), and then underlying GPU drivers are
>> just one category of users of this general vdev framework. In the
>> future it's easily extended to support other I/O virtualization based 
>> on similar vgpu concept;
>>
>> Second, are these 3 patches already working with nvidia device,
>> or are they just conceptual implementation w/o completing actual
>> test yet? We'll start moving our implementation toward this direction
>> too, so would be good to know the current status and how we can
>> further cooperate to move forward. Based on that we can start 
>> giving more comments on next level detail.
>>
> 
> Hi Kevin,
> 
> Yes, we do have an engineering prototype up and running with this set of 
> kernel
> patches we have posted.
> 

Good to know that :)

> Please let us know if you have any questions while integrating your vgpu 
> solution
> within this framework.

Thanks for your work, we are evaluating the integrate of the framework
with our vgpu implementation, will make/propose changes to this.

> 
> Thanks,
> Neo
> 
--
Thanks,
Jike

Re: [Qemu-devel] [RFC PATCH v2 1/3] vGPU Core driver

[Neo Jia]

On Mon, Feb 29, 2016 at 05:39:02AM +, Tian, Kevin wrote:
> > From: Kirti Wankhede
> > Sent: Wednesday, February 24, 2016 12:24 AM
> > 
> > Signed-off-by: Kirti Wankhede 
> > Signed-off-by: Neo Jia 
> 
> Hi, Kirti/Neo,
> 
> Thanks a lot for you updated version. Having not looked into detail
> code, first come with some high level comments.
> 
> First, in a glimpse the majority of the code (possibly >95%) is device
> agnostic, though we call it vgpu today. Just thinking about the
> extensibility and usability of this framework, would it be better to 
> name it in a way that any other type of I/O device can be fit into 
> this framework? I don't have a good idea of the name now, but 
> a simple idea is to replace vgpu with vdev (vdev-core, vfio-vdev,
> vfio-iommu-type1-vdev, etc.), and then underlying GPU drivers are
> just one category of users of this general vdev framework. In the
> future it's easily extended to support other I/O virtualization based 
> on similar vgpu concept;
> 
> Second, are these 3 patches already working with nvidia device,
> or are they just conceptual implementation w/o completing actual
> test yet? We'll start moving our implementation toward this direction
> too, so would be good to know the current status and how we can
> further cooperate to move forward. Based on that we can start 
> giving more comments on next level detail.
> 

Hi Kevin,

Yes, we do have an engineering prototype up and running with this set of kernel
patches we have posted.

Please let us know if you have any questions while integrating your vgpu 
solution
within this framework.

Thanks,
Neo

> Thanks
> Kevin

Re: [Qemu-devel] [RFC PATCH v2 1/3] vGPU Core driver

[Tian, Kevin]

> From: Kirti Wankhede
> Sent: Wednesday, February 24, 2016 12:24 AM
> 
> Design for vGPU Driver:
> Main purpose of vGPU driver is to provide a common interface for vGPU
> management that can be used by differnt GPU drivers.
> 
> This module would provide a generic interface to create the device, add
> it to vGPU bus, add device to IOMMU group and then add it to vfio group.
> 
> High Level block diagram:
> 
> +--+vgpu_register_driver()+---+
> | __init() +->+   |
> |  |  |   |
> |  +<-+vgpu.ko|
> | vgpu_vfio.ko |   probe()/remove()   |   |
> |  |+-+   +-+
> +--+| +---+---+ |
> | ^ |
> | callback| |
> | +---++|
> | |vgpu_register_device()   |
> | |||
> +---^-+-++-+--+-+
> | nvidia.ko ||  i915.ko   |
> |   |||
> +---+++
> 
> vGPU driver provides two types of registration interfaces:
> 1. Registration interface for vGPU bus driver:
> 
> /**
>   * struct vgpu_driver - vGPU device driver
>   * @name: driver name
>   * @probe: called when new device created
>   * @remove: called when device removed
>   * @driver: device driver structure
>   *
>   **/
> struct vgpu_driver {
>  const char *name;
>  int  (*probe)  (struct device *dev);
>  void (*remove) (struct device *dev);
>  struct device_driverdriver;
> };
> 
> int  vgpu_register_driver(struct vgpu_driver *drv, struct module *owner);
> void vgpu_unregister_driver(struct vgpu_driver *drv);
> 
> VFIO bus driver for vgpu, should use this interface to register with
> vGPU driver. With this, VFIO bus driver for vGPU devices is responsible
> to add vGPU device to VFIO group.
> 
> 2. GPU driver interface
> GPU driver interface provides GPU driver the set APIs to manage GPU driver
> related work in their own driver. APIs are to:
> - vgpu_supported_config: provide supported configuration list by the GPU.
> - vgpu_create: to allocate basic resouces in GPU driver for a vGPU device.
> - vgpu_destroy: to free resources in GPU driver during vGPU device destroy.
> - vgpu_start: to initiate vGPU initialization process from GPU driver when VM
>   boots and before QEMU starts.
> - vgpu_shutdown: to teardown vGPU resources during VM teardown.
> - read : read emulation callback.
> - write: write emulation callback.
> - vgpu_set_irqs: send interrupt configuration information that QEMU sets.
> - vgpu_bar_info: to provice BAR size and its flags for the vGPU device.
> - validate_map_request: to validate remap pfn request.
> 
> This registration interface should be used by GPU drivers to register
> each physical device to vGPU driver.
> 
> Updated this patch with couple of more functions in GPU driver interface
> which were discussed during v1 version of this RFC.
> 
> Thanks,
> Kirti.
> 
> Signed-off-by: Kirti Wankhede 
> Signed-off-by: Neo Jia 

Hi, Kirti/Neo,

Thanks a lot for you updated version. Having not looked into detail
code, first come with some high level comments.

First, in a glimpse the majority of the code (possibly >95%) is device
agnostic, though we call it vgpu today. Just thinking about the
extensibility and usability of this framework, would it be better to 
name it in a way that any other type of I/O device can be fit into 
this framework? I don't have a good idea of the name now, but 
a simple idea is to replace vgpu with vdev (vdev-core, vfio-vdev,
vfio-iommu-type1-vdev, etc.), and then underlying GPU drivers are
just one category of users of this general vdev framework. In the
future it's easily extended to support other I/O virtualization based 
on similar vgpu concept;

Second, are these 3 patches already working with nvidia device,
or are they just conceptual implementation w/o completing actual
test yet? We'll start moving our implementation toward this direction
too, so would be good to know the current status and how we can
further cooperate to move forward. Based on that we can start 
giving more comments on next level detail.

Thanks
Kevin

[Qemu-devel] [RFC PATCH v2 1/3] vGPU Core driver

[Kirti Wankhede]

Design for vGPU Driver:
Main purpose of vGPU driver is to provide a common interface for vGPU
management that can be used by differnt GPU drivers.

This module would provide a generic interface to create the device, add
it to vGPU bus, add device to IOMMU group and then add it to vfio group.

High Level block diagram:

+--+vgpu_register_driver()+---+
| __init() +->+   |
|  |  |   |
|  +<-+vgpu.ko|
| vgpu_vfio.ko |   probe()/remove()   |   |
|  |+-+   +-+
+--+| +---+---+ |
| ^ |
| callback| |
| +---++|
| |vgpu_register_device()   |
| |||
+---^-+-++-+--+-+
| nvidia.ko ||  i915.ko   |
|   |||
+---+++

vGPU driver provides two types of registration interfaces:
1. Registration interface for vGPU bus driver:

/**
  * struct vgpu_driver - vGPU device driver
  * @name: driver name
  * @probe: called when new device created
  * @remove: called when device removed
  * @driver: device driver structure
  *
  **/
struct vgpu_driver {
 const char *name;
 int  (*probe)  (struct device *dev);
 void (*remove) (struct device *dev);
 struct device_driverdriver;
};

int  vgpu_register_driver(struct vgpu_driver *drv, struct module *owner);
void vgpu_unregister_driver(struct vgpu_driver *drv);

VFIO bus driver for vgpu, should use this interface to register with
vGPU driver. With this, VFIO bus driver for vGPU devices is responsible
to add vGPU device to VFIO group.

2. GPU driver interface
GPU driver interface provides GPU driver the set APIs to manage GPU driver
related work in their own driver. APIs are to:
- vgpu_supported_config: provide supported configuration list by the GPU.
- vgpu_create: to allocate basic resouces in GPU driver for a vGPU device.
- vgpu_destroy: to free resources in GPU driver during vGPU device destroy.
- vgpu_start: to initiate vGPU initialization process from GPU driver when VM
  boots and before QEMU starts.
- vgpu_shutdown: to teardown vGPU resources during VM teardown.
- read : read emulation callback.
- write: write emulation callback.
- vgpu_set_irqs: send interrupt configuration information that QEMU sets.
- vgpu_bar_info: to provice BAR size and its flags for the vGPU device.
- validate_map_request: to validate remap pfn request.

This registration interface should be used by GPU drivers to register
each physical device to vGPU driver.

Updated this patch with couple of more functions in GPU driver interface
which were discussed during v1 version of this RFC.

Thanks,
Kirti.

Signed-off-by: Kirti Wankhede 
Signed-off-by: Neo Jia 
---
 drivers/Kconfig |2 +
 drivers/Makefile|1 +
 drivers/vgpu/Kconfig|   26 +++
 drivers/vgpu/Makefile   |4 +
 drivers/vgpu/vgpu-core.c|  422 +++
 drivers/vgpu/vgpu-driver.c  |  137 ++
 drivers/vgpu/vgpu-sysfs.c   |  366 +
 drivers/vgpu/vgpu_private.h |   36 
 include/linux/vgpu.h|  217 ++
 9 files changed, 1211 insertions(+), 0 deletions(-)
 create mode 100644 drivers/vgpu/Kconfig
 create mode 100644 drivers/vgpu/Makefile
 create mode 100644 drivers/vgpu/vgpu-core.c
 create mode 100644 drivers/vgpu/vgpu-driver.c
 create mode 100644 drivers/vgpu/vgpu-sysfs.c
 create mode 100644 drivers/vgpu/vgpu_private.h
 create mode 100644 include/linux/vgpu.h

diff --git a/drivers/Kconfig b/drivers/Kconfig
index d2ac339..5fd9eae 100644
--- a/drivers/Kconfig
+++ b/drivers/Kconfig
@@ -122,6 +122,8 @@ source "drivers/uio/Kconfig"
 
 source "drivers/vfio/Kconfig"
 
+source "drivers/vgpu/Kconfig"
+
 source "drivers/vlynq/Kconfig"
 
 source "drivers/virt/Kconfig"
diff --git a/drivers/Makefile b/drivers/Makefile
index 795d0ca..1c43250 100644
--- a/drivers/Makefile
+++ b/drivers/Makefile
@@ -84,6 +84,7 @@ obj-$(CONFIG_FUSION)  += message/
 obj-y  += firewire/
 obj-$(CONFIG_UIO)  += uio/
 obj-$(CONFIG_VFIO) += vfio/
+obj-$(CONFIG_VFIO) += vgpu/
 obj-y  += cdrom/
 obj-y  += auxdisplay/
 obj-$(CONFIG_PCCARD)   += pcmcia/
diff --git a/drivers/vgpu/Kconfig