Re: [Qemu-devel] [RFC PATCH v1 1/1] vGPU core driver : to provide common interface for vGPU.
On Wed, Feb 17, 2016 at 02:08:18PM +0100, Gerd Hoffmann wrote: > Hi, > > > For example, how to locate the path of a given VM? > > You go ask libvirt, the domain xml will have the info. > > > Whoever is going to configure > > the qemu has to walk through *all* the current vgpu path to locate the UUID > > to > > match the QEMU's VM UUID. > > No. qemu simply uses the path it get passed from libvirt. libvirt > simply uses whatever is stored in the domain xml. > > i.e. you'll have a config like this: > > > rhel7-vfio > 0990b05d-4fbd-49bf-88e4-e87974c64fba > [ ... ] > > [ ... ] > > > > > bus='0x00' slot='0x04' function='0x0'/> > > > > I think I has answered this, UUID is not a user space or kernel space > > concept, it is just a generic way to represent object, > > Yes. But the above sounds like you want to use UUIDs to *link* two > objects, by assigning the same uuid to both vm and vgpu. This is *not* > how uuids should be used. Each object should have its own uuid. > > You can use uuids to name the vgpus if you want of course. But the vgpu > uuid will will have no relationship whatsoever to the vm uuid then. > Agree. I should made it clear that it should be a separate object. Thanks, Neo > cheers, > Gerd >
Re: [Qemu-devel] [RFC PATCH v1 1/1] vGPU core driver : to provide common interface for vGPU.
Hi, > For example, how to locate the path of a given VM? You go ask libvirt, the domain xml will have the info. > Whoever is going to configure > the qemu has to walk through *all* the current vgpu path to locate the UUID to > match the QEMU's VM UUID. No. qemu simply uses the path it get passed from libvirt. libvirt simply uses whatever is stored in the domain xml. i.e. you'll have a config like this: rhel7-vfio 0990b05d-4fbd-49bf-88e4-e87974c64fba [ ... ] [ ... ] > I think I has answered this, UUID is not a user space or kernel space > concept, it is just a generic way to represent object, Yes. But the above sounds like you want to use UUIDs to *link* two objects, by assigning the same uuid to both vm and vgpu. This is *not* how uuids should be used. Each object should have its own uuid. You can use uuids to name the vgpus if you want of course. But the vgpu uuid will will have no relationship whatsoever to the vm uuid then. cheers, Gerd
Re: [Qemu-devel] [RFC PATCH v1 1/1] vGPU core driver : to provide common interface for vGPU.
> From: Neo Jia [mailto:c...@nvidia.com] > Sent: Wednesday, February 17, 2016 6:34 PM > > > > > Qemu is not a kernel component. And UUID is OPTIONAL for Qemu. > > > > KVM is the kernel component. It doesn't use UUID at all. the relation > > between > > UUID and VM is fully maintained in user space. > > Hold on ... we are talking about the vgpu.ko not KVM right? You said UUID is a general way to represent an object. VM is an object. There is UUID associated with a VM. But UUID is not recorded in the kernel KVM. It's a good reference to our design on vgpu object. If one thing can be done in user space, then better not involve kernel side. > > > > > > This is why I think we should use this UUID as a generic management > > > interface, > > > and we shouldn't have anything optional. > > > > > > > I don't buy-in this argument. I always think kernel design should provide > > enough flexibility, instead of assuming user space behavior. > > > > I think you are using the wrong terms here. Flexibility doesn't apply here. > What > we are trying to achieve here is to have a generic interface for upper layer > software to manage vgpu device. Even Qemu doesn't make UUID as mandatory. It's just an optional parameter. I'm not sure why you insist UUID mandatory for vGPU here. > > > Let me also add some Citrix friends. See how they feel about the necessity > > of > > having UUID in vgpu name. > > Sorry? Paul/Malcolm are experts in Citrix XenServer. So I think they may give some suggestions from their angle. :-) Thanks Kevin
Re: [Qemu-devel] [RFC PATCH v1 1/1] vGPU core driver : to provide common interface for vGPU.
On Wed, Feb 17, 2016 at 09:52:04AM +, Tian, Kevin wrote: > > From: Neo Jia [mailto:c...@nvidia.com] > > Sent: Wednesday, February 17, 2016 5:35 PM > > > > On Wed, Feb 17, 2016 at 08:57:08AM +, Tian, Kevin wrote: > > > > From: Neo Jia [mailto:c...@nvidia.com] > > > > Sent: Wednesday, February 17, 2016 3:55 PM > > > > > If we can make it free, why not? > > I can buy-in this argument. Great! > > Qemu is not a kernel component. And UUID is OPTIONAL for Qemu. > > KVM is the kernel component. It doesn't use UUID at all. the relation between > UUID and VM is fully maintained in user space. Hold on ... we are talking about the vgpu.ko not KVM right? UUID is just a generic way to represent an object, here we use a uuid to represent a virtual gpu device. > > > > > Please also note that using UUID to represent a virtual gpu device directory > > doesn't mean UUID is part of a GPU resource. > > but it adds a hard dependency on another resource - UUID. > > > > > > > > > So let's keep UUID as an optional parameter. When UUID is provided, it > > > will be included in the vGPU name then your requirement can be met. > > > > > > > Like I have said before, we are seeking a generic interface to allow upper > > layer > > software stack to manage vgpu device for different vendors, so we should > > not really > > consider "an optional path for vgpu device discovery" at all. > > > > This is why I think we should use this UUID as a generic management > > interface, > > and we shouldn't have anything optional. > > > > I don't buy-in this argument. I always think kernel design should provide > enough flexibility, instead of assuming user space behavior. > I think you are using the wrong terms here. Flexibility doesn't apply here. What we are trying to achieve here is to have a generic interface for upper layer software to manage vgpu device. > Let me also add some Citrix friends. See how they feel about the necessity of > having UUID in vgpu name. Sorry? Thanks, Neo > > Thanks > Kevin
Re: [Qemu-devel] [RFC PATCH v1 1/1] vGPU core driver : to provide common interface for vGPU.
> From: Neo Jia [mailto:c...@nvidia.com] > Sent: Wednesday, February 17, 2016 5:35 PM > > On Wed, Feb 17, 2016 at 08:57:08AM +, Tian, Kevin wrote: > > > From: Neo Jia [mailto:c...@nvidia.com] > > > Sent: Wednesday, February 17, 2016 3:55 PM > > > > 'whoever' is too strict here. I don't think UUID is required in all > > scenarios. > > > > In your scenario: > > > > - You will pass VM UUID when creating a vgpu. > > - Consequently a /sys/device/virtual/vgpu/$UUID-$vgpu-id is created > > - Then you can identify $UUID-$vgpu-id is right for the very VM, by matching > > all available vgpu nodes with VM UUID; > > > > When it is a bit convenient, I don't see it significant. Looping directory > > is > > not unusual for file/directory operations and it happens infrequently only > > for vgpu life-cycle mgmt.. > > Hi Kevin, > > The search is expensive, when you have 8 physical gpus and each can support up > to 32 vgpus. vgpu life-cycle management happens a lot as well in real > production > scenario. > > If we can make it free, why not? I can buy-in this argument. > > > > > Please think about my original proposal carefully. I'm not opposing encoding > > UUID in vgpu name. What I'm opposing is not to make it mandatory, i.e. when > > UUID is not provided, we should still allow vgpu creation using some default > > descriptive string. > > Probably you are not quite get the generic design that we are proposing here. > > The goal here is to have a unified interface for all gpu vendor, and expose > that > to the upper layer software stack, so I don't think we should have an optional > vgpu device discovery path at all. > > If we have an optional case, does that mean libvirt will have a different > implementation and qemu will have a different implementation? I don't think > that > is acceptable. libvirt can choose to always provide UUID, then there's no problem. Another stack can choose to not use UUID. It doesn't mean each stack needs to support different implementation. Just from kernel p.o.v we need sustain such flexibility. Qemu will always have the same implementation. I explained this multiple times. Qemu only cares about a sysfsdev path parameter. It doesn't matter whether it has a UUID included or not. > > Since you have admitted this design is convenient and performance better, I > think we > should stay with it. > > > > > 'user space dependency' means you need additional user-space operations > > (say uuidgen here) before you can utilize GPU virtualization feature, which > > is not necessary. In reality, UUID is not a GPU resource. It's not what GPU > > virtualization intrinsically needs to handle. Let's keep vGPU-core > > sub-system > > modulo for its real functionalities. > > Don't you need to create UUID to make qemu happy? I don't get this argument. Qemu is not a kernel component. And UUID is OPTIONAL for Qemu. KVM is the kernel component. It doesn't use UUID at all. the relation between UUID and VM is fully maintained in user space. > > Please also note that using UUID to represent a virtual gpu device directory > doesn't mean UUID is part of a GPU resource. but it adds a hard dependency on another resource - UUID. > > > > > So let's keep UUID as an optional parameter. When UUID is provided, it > > will be included in the vGPU name then your requirement can be met. > > > > Like I have said before, we are seeking a generic interface to allow upper > layer > software stack to manage vgpu device for different vendors, so we should not > really > consider "an optional path for vgpu device discovery" at all. > > This is why I think we should use this UUID as a generic management interface, > and we shouldn't have anything optional. > I don't buy-in this argument. I always think kernel design should provide enough flexibility, instead of assuming user space behavior. Let me also add some Citrix friends. See how they feel about the necessity of having UUID in vgpu name. Thanks Kevin
Re: [Qemu-devel] [RFC PATCH v1 1/1] vGPU core driver : to provide common interface for vGPU.
On Wed, Feb 17, 2016 at 08:57:08AM +, Tian, Kevin wrote: > > From: Neo Jia [mailto:c...@nvidia.com] > > Sent: Wednesday, February 17, 2016 3:55 PM > > 'whoever' is too strict here. I don't think UUID is required in all scenarios. > > In your scenario: > > - You will pass VM UUID when creating a vgpu. > - Consequently a /sys/device/virtual/vgpu/$UUID-$vgpu-id is created > - Then you can identify $UUID-$vgpu-id is right for the very VM, by matching > all available vgpu nodes with VM UUID; > > When it is a bit convenient, I don't see it significant. Looping directory is > not unusual for file/directory operations and it happens infrequently only > for vgpu life-cycle mgmt.. Hi Kevin, The search is expensive, when you have 8 physical gpus and each can support up to 32 vgpus. vgpu life-cycle management happens a lot as well in real production scenario. If we can make it free, why not? > > Please think about my original proposal carefully. I'm not opposing encoding > UUID in vgpu name. What I'm opposing is not to make it mandatory, i.e. when > UUID is not provided, we should still allow vgpu creation using some default > descriptive string. Probably you are not quite get the generic design that we are proposing here. The goal here is to have a unified interface for all gpu vendor, and expose that to the upper layer software stack, so I don't think we should have an optional vgpu device discovery path at all. If we have an optional case, does that mean libvirt will have a different implementation and qemu will have a different implementation? I don't think that is acceptable. Since you have admitted this design is convenient and performance better, I think we should stay with it. > > 'user space dependency' means you need additional user-space operations > (say uuidgen here) before you can utilize GPU virtualization feature, which > is not necessary. In reality, UUID is not a GPU resource. It's not what GPU > virtualization intrinsically needs to handle. Let's keep vGPU-core sub-system > modulo for its real functionalities. Don't you need to create UUID to make qemu happy? I don't get this argument. Please also note that using UUID to represent a virtual gpu device directory doesn't mean UUID is part of a GPU resource. > > So let's keep UUID as an optional parameter. When UUID is provided, it > will be included in the vGPU name then your requirement can be met. > Like I have said before, we are seeking a generic interface to allow upper layer software stack to manage vgpu device for different vendors, so we should not really consider "an optional path for vgpu device discovery" at all. This is why I think we should use this UUID as a generic management interface, and we shouldn't have anything optional. Thanks, Neo > Thanks > Kevin
Re: [Qemu-devel] [RFC PATCH v1 1/1] vGPU core driver : to provide common interface for vGPU.
> From: Neo Jia [mailto:c...@nvidia.com] > Sent: Wednesday, February 17, 2016 4:41 PM > > On Wed, Feb 17, 2016 at 07:51:12AM +, Tian, Kevin wrote: > > > From: Neo Jia [mailto:c...@nvidia.com] > > > Sent: Wednesday, February 17, 2016 3:32 PM > > > > > > On Wed, Feb 17, 2016 at 07:52:53AM +0100, Gerd Hoffmann wrote: > > > > Hi, > > > > > > > > > The answer is simple, having a UUID as part of the device name will > > > > > give you a > > > > > unique sysfs path that will be opened by QEMU. > > > > > > > > A descriptive name will work too, and I think it'll be easier to make > > > > those names persistent because you don't have to store the uuids > > > > somewhere to re-create the same setup afer reboot. > > > > > > Hi Gerd, > > > > > > Right, UUID will be persistent cross reboot. The qemu vgpu path for a > > > given VM will > > > not get changed when it gets reboots and multiple other devices have been > > > created in the middle. > > > > Curious why persistence matters here. It's completely OK to assign > > another vgpu to this VM across reboot, as long as the new vgpu > > provides same capability to previous one. > > Hi Kevin, > > Those virtual devices might be destroyed and re-created as part of the reboot > or > shutdown. The user doesn't want to change his configuration, if the > path is not associated with UUID, the user might get a different vgpu than his > previous configuration, unpredictable. As long as libvirt keeps configuration under UUID, and then update UUID-> vgpu_path correctly, it won't be a problem: UUID: config1, config2, reboot1: UUID -> vgpu1 reboot2: UUID -> vgpu2 Then you can always have the configuration applied to new vgpu correctly. Just want to state it's not a hard limitation. But as I replied in another mail, I'm not against including UUID in the name, but please keep the option for UUID not being used. Thanks Kevin
Re: [Qemu-devel] [RFC PATCH v1 1/1] vGPU core driver : to provide common interface for vGPU.
> From: Neo Jia [mailto:c...@nvidia.com] > Sent: Wednesday, February 17, 2016 3:55 PM > > On Wed, Feb 17, 2016 at 07:46:15AM +, Tian, Kevin wrote: > > > From: Neo Jia > > > Sent: Wednesday, February 17, 2016 3:26 PM > > > > > > > > > > > > > > If your most concern is having this kind of path doesn't provide enough > > > information of the virtual device, we can add more sysfs attributes > > > within the > > > directory of /sys/devices/virtual/vgpu/$UUID-$vgpu_idx/ to reflect the > > > information you want. > > > > Like Gerd said, you can have something like this: > > > > -device vfio-pci,sysfsdev=/sys/devices/virtual/vgpu/vgpu_idx/UUID > > Hi Kevin, > > The vgpu_idx is not unique number at all. > > For example, how to locate the path of a given VM? Whoever is going to > configure > the qemu has to walk through *all* the current vgpu path to locate the UUID to > match the QEMU's VM UUID. This is not required if you have UUID as part of the > device path. 'whoever' is too strict here. I don't think UUID is required in all scenarios. In your scenario: - You will pass VM UUID when creating a vgpu. - Consequently a /sys/device/virtual/vgpu/$UUID-$vgpu-id is created - Then you can identify $UUID-$vgpu-id is right for the very VM, by matching all available vgpu nodes with VM UUID; When it is a bit convenient, I don't see it significant. Looping directory is not unusual for file/directory operations and it happens infrequently only for vgpu life-cycle mgmt.. Please think about my original proposal carefully. I'm not opposing encoding UUID in vgpu name. What I'm opposing is not to make it mandatory, i.e. when UUID is not provided, we should still allow vgpu creation using some default descriptive string. > > > > > > > > > Even with UUID, you don't need libvirt at all. you can get uuid by running > > > uuidgen command, I don't need libvirt to code up and test the RFC that I > > > have > > > sent out early. :-) > > > > although simple, it still creates unnecessary user space dependency for > > kernel resource management... > > I think I has answered this, UUID is not a user space or kernel space > concept, it is just a generic way to represent object, it just make sure that > virtual gpu device directory can be uniquely addressed. > 'user space dependency' means you need additional user-space operations (say uuidgen here) before you can utilize GPU virtualization feature, which is not necessary. In reality, UUID is not a GPU resource. It's not what GPU virtualization intrinsically needs to handle. Let's keep vGPU-core sub-system modulo for its real functionalities. So let's keep UUID as an optional parameter. When UUID is provided, it will be included in the vGPU name then your requirement can be met. Thanks Kevin
Re: [Qemu-devel] [RFC PATCH v1 1/1] vGPU core driver : to provide common interface for vGPU.
On Wed, Feb 17, 2016 at 07:51:12AM +, Tian, Kevin wrote: > > From: Neo Jia [mailto:c...@nvidia.com] > > Sent: Wednesday, February 17, 2016 3:32 PM > > > > On Wed, Feb 17, 2016 at 07:52:53AM +0100, Gerd Hoffmann wrote: > > > Hi, > > > > > > > The answer is simple, having a UUID as part of the device name will > > > > give you a > > > > unique sysfs path that will be opened by QEMU. > > > > > > A descriptive name will work too, and I think it'll be easier to make > > > those names persistent because you don't have to store the uuids > > > somewhere to re-create the same setup afer reboot. > > > > Hi Gerd, > > > > Right, UUID will be persistent cross reboot. The qemu vgpu path for a given > > VM will > > not get changed when it gets reboots and multiple other devices have been > > created in the middle. > > Curious why persistence matters here. It's completely OK to assign > another vgpu to this VM across reboot, as long as the new vgpu > provides same capability to previous one. Hi Kevin, Those virtual devices might be destroyed and re-created as part of the reboot or shutdown. The user doesn't want to change his configuration, if the path is not associated with UUID, the user might get a different vgpu than his previous configuration, unpredictable. We can't change the vgpu configuration behind uses back, especially we have a lot of physical devices and tons virtual gpus can be assigned. Also, if user wants to move the vm configuration to a different host, it is more nature to decouple the vgpu configuration from VM itself. So the VM will only open up the device addressed by the corresponding UUID, what you need is just to describe his vgpu, and it doesn't matter how you describe and doesn't matter how many virtual devices have been created already on that physical device, your QEMU path is always persistent. Thanks, Neo > > Thanks > Kevin
Re: [Qemu-devel] [RFC PATCH v1 1/1] vGPU core driver : to provide common interface for vGPU.
On Wed, Feb 17, 2016 at 07:46:15AM +, Tian, Kevin wrote: > > From: Neo Jia > > Sent: Wednesday, February 17, 2016 3:26 PM > > > > > > > > > If your most concern is having this kind of path doesn't provide enough > > information of the virtual device, we can add more sysfs attributes within > > the > > directory of /sys/devices/virtual/vgpu/$UUID-$vgpu_idx/ to reflect the > > information you want. > > Like Gerd said, you can have something like this: > > -device vfio-pci,sysfsdev=/sys/devices/virtual/vgpu/vgpu_idx/UUID Hi Kevin, The vgpu_idx is not unique number at all. For example, how to locate the path of a given VM? Whoever is going to configure the qemu has to walk through *all* the current vgpu path to locate the UUID to match the QEMU's VM UUID. This is not required if you have UUID as part of the device path. > > > > > Even with UUID, you don't need libvirt at all. you can get uuid by running > > uuidgen command, I don't need libvirt to code up and test the RFC that I > > have > > sent out early. :-) > > although simple, it still creates unnecessary user space dependency for > kernel resource management... I think I has answered this, UUID is not a user space or kernel space concept, it is just a generic way to represent object, it just make sure that virtual gpu device directory can be uniquely addressed. Thanks, Neo > > Thanks > Kevin
Re: [Qemu-devel] [RFC PATCH v1 1/1] vGPU core driver : to provide common interface for vGPU.
> From: Neo Jia [mailto:c...@nvidia.com] > Sent: Wednesday, February 17, 2016 3:32 PM > > On Wed, Feb 17, 2016 at 07:52:53AM +0100, Gerd Hoffmann wrote: > > Hi, > > > > > The answer is simple, having a UUID as part of the device name will give > > > you a > > > unique sysfs path that will be opened by QEMU. > > > > A descriptive name will work too, and I think it'll be easier to make > > those names persistent because you don't have to store the uuids > > somewhere to re-create the same setup afer reboot. > > Hi Gerd, > > Right, UUID will be persistent cross reboot. The qemu vgpu path for a given > VM will > not get changed when it gets reboots and multiple other devices have been > created in the middle. Curious why persistence matters here. It's completely OK to assign another vgpu to this VM across reboot, as long as the new vgpu provides same capability to previous one. Thanks Kevin
Re: [Qemu-devel] [RFC PATCH v1 1/1] vGPU core driver : to provide common interface for vGPU.
> From: Neo Jia > Sent: Wednesday, February 17, 2016 3:26 PM > > > > > > Qemu doesn't need to know the relation between virtual/physical devices at > > all. It's just a path regardless of how vgpu name is created (either with > > your > > UUID proposal or my descriptive string proposal) > > No, with path like above, QEMU needs to know the virtual device is created > from > that physical device :00:02.0 right? (you have mentioned this yourself > actually below.) If QEMU doesn't want to know that, then he will transfer the I didn't say anything that Qemu needs to know it. I said "I can immediately know ..." which means a human being. :-) Qemu only needs a path to open. [...] > > > > > This is a typical way how device nodes are created within sysfs, e.g. on my > > platform: > > > > $ ls /sys/class/drm/ > > card0/ card0-DP-2/ card0-HDMI-A-2/ controlD64/ > > card0-DP-1/ card0-HDMI-A-1/ card0-VGA-1/version > > > > $ ls /sys/bus/pci/devices/ > > :00:00.0 :00:14.0 :00:1a.0 :00:1c.1 :00:1f.2 > > :00:02.0 :00:16.0 :00:1b.0 :00:1d.0 :00:1f.3 > > :00:03.0 :00:19.0 :00:1c.0 :00:1f.0 :02:00.0 > > > > We'd better keep such style when creating vgpu nodes in sysfs. UUID is > > at most anther info suffixed to the default string (or in another file), if > > necessary. > > It doesn't apply here, your above example are all physical devices. > > The reason I want to have UUID is it to match the instance of problem domain > here - VM. It doesn't matter whether it's physical or virtual. Sysfs includes many nodes not physically existing. The point that I don't understand, is why you insist the only way to associate vgpu to a VM is by encoding UUID in vgpu name. libvirt maintains many attributes (including other virtual devices) for a given VM, in its internal database. It's not a problem to reverse find a VM according to a general vgpu name. [...] > > > > I'm fine to have another node to provide more vendor specific information. > > But > > I don't want to make UUID mandatory when creating a vGPU instance, as > > explained above. Today we can create VMs in KVM w/o using libvirt, and w/o > > the need of allocating any UUID. Same thing should be supported for vgpu > > feature too. > > So, I really don't see any drawback of using UUID as part of the virtual > device > directory, it is easy, simple and organically reflecting the relation between > virtual device and the owner. "organically reflecting" only when other database is included (as libvirt is involved), while losing the merit which a descriptive name can bring. > > Each QEMU process is representing a VM, and a UUID is associated with it. The > virtual gpu device is just a virtual device of this owner, so the path is: > > -device vfio-pci,sysfsdev=/sys/devices/virtual/vgpu/$UUID-$vgpu_idx/ > > You can have multiple virtual device per VM and QEMU doesn't need to know > which > physical device it comes from, especially it will automatically know which > virtual device it owns, so the -device vfio-pci path will be setup for free. As I commented earlier, Qemu never needs to know that information regardless of how the vgpu is named. > > If your most concern is having this kind of path doesn't provide enough > information of the virtual device, we can add more sysfs attributes within the > directory of /sys/devices/virtual/vgpu/$UUID-$vgpu_idx/ to reflect the > information you want. Like Gerd said, you can have something like this: -device vfio-pci,sysfsdev=/sys/devices/virtual/vgpu/vgpu_idx/UUID > > Even with UUID, you don't need libvirt at all. you can get uuid by running > uuidgen command, I don't need libvirt to code up and test the RFC that I have > sent out early. :-) although simple, it still creates unnecessary user space dependency for kernel resource management... Thanks Kevin
Re: [Qemu-devel] [RFC PATCH v1 1/1] vGPU core driver : to provide common interface for vGPU.
On Wed, Feb 17, 2016 at 07:52:53AM +0100, Gerd Hoffmann wrote: > Hi, > > > The answer is simple, having a UUID as part of the device name will give > > you a > > unique sysfs path that will be opened by QEMU. > > A descriptive name will work too, and I think it'll be easier to make > those names persistent because you don't have to store the uuids > somewhere to re-create the same setup afer reboot. Hi Gerd, Right, UUID will be persistent cross reboot. The qemu vgpu path for a given VM will not get changed when it gets reboots and multiple other devices have been created in the middle. > > > If you are worried about losing meaningful name here, we can create a sysfs > > file > > to capture the vendor device description if you like. > > You can also store the uuid in a sysfs file ... Another benefit is that having the UUID as part of the virtual vgpu device path will allow whoever is going to config the QEMU to automatically discover the virtual device sysfs for free. Thanks, Neo > > cheers, > Gerd >
Re: [Qemu-devel] [RFC PATCH v1 1/1] vGPU core driver : to provide common interface for vGPU.
On Wed, Feb 17, 2016 at 06:02:36AM +, Tian, Kevin wrote: > > From: Neo Jia > > Sent: Wednesday, February 17, 2016 1:38 PM > > > > > > > > > > > > > > > > > > Hi Kevin, > > > > > > > > The answer is simple, having a UUID as part of the device name will > > > > give you a > > > > unique sysfs path that will be opened by QEMU. > > > > > > > > vgpu-vendor-0 and vgpu-vendor-1 will not be unique as we can have > > > > multiple > > > > virtual gpu devices per VM coming from same or different physical > > > > devices. > > > > > > That is not a problem. We can add physical device info too like > > > vgpu-vendor-0-0, > > > vgpu-vendor-1-0, ... > > > > > > Please note Qemu doesn't care about the actual name. It just accepts a > > > sysfs path > > > to open. > > > > Hi Kevin, > > > > No, I think you are making things even more complicated than it is required, > > also it is not generic anymore as you are requiring the QEMU to know more > > than > > he needs to. > > > > The way you name those devices will require QEMU to know the relation > > between virtual devices and physical devices. I don't think that is good. > > I don't think you get my point. Look at how device is assigned in Qemu today: > > -device vfio-pci,host=02:00.0 > > Then in a recent patch from Alex, Qemu will accept sysfsdev as well: > > -device vfio-pci,sysfsdev=/sys/devices/pci:00/:00:1c.0/:02:00.0 > > Then with vgu (one example from Alex's original post): > > -device vfio-pci,sysfsdev=/sys/devices/virtual/intel-vgpu/vgpu0@:00:02.0 Hi Kevin, I am fully aware of Alex's patch, that is just an example, but he doesn't exclude the cases of using UUID as the device path as he mentioned in the beginning of this long email thread, actually he has already agreed with this UUID-$vgpu_idx path. :-) Also, you should note that with the proposal we have, it doesn't require to have anything like either intel-vgpu or nvidia-vgpu as the path, having a non-vendor specific information within the vgpu device path is one of the requirement IIRC. -device vfio-pci,sysfsdev=/sys/devices/virtual/intel-vgpu/vgpu0@:00:02.0 > > Qemu doesn't need to know the relation between virtual/physical devices at > all. It's just a path regardless of how vgpu name is created (either with your > UUID proposal or my descriptive string proposal) No, with path like above, QEMU needs to know the virtual device is created from that physical device :00:02.0 right? (you have mentioned this yourself actually below.) If QEMU doesn't want to know that, then he will transfer the burden to the upper layer stack such as libvirt, who has to figure out the right path of this new VM, as the vgpu<$id>, the <%id> will become another generic number generated by the vgpu core driver. So why not just use UUID here? > > > > > My flow is like this: > > > > libvirt creats a VM object, it will have a UUID. then it will use the UUID > > to > > create virtual gpu devices, then it will pass the UUID to the QEMU (actually > > QEMU already has the VM UUID), then it will just open up the unique path. > > If you look at above example, it's not UUID itself being passed to Qemu. It's > the sysfsdev path. UUID is always sent to QEMU, if you look at your QEMU command line, please. Yes, it is a sysfs path, even with UUID, it is a sysfs path, the only difference is that we have a uuid embedded within the device name. > > > > > Also, you need to consider those 0-0 numbers are not generic as the UUID. > > that encoding could be flexible to include any meaningful string. libvirt can > itself manages how UUID is mapped to an actual vgpu name. > > > > > > > > > > > > If you are worried about losing meaningful name here, we can create a > > > > sysfs file > > > > to capture the vendor device description if you like. > > > > > > > > > > Having the vgpu name descriptive is more informative imo. User can simply > > > check > > > sysfs names to know raw information w/o relying on 3rd party agent to > > > query > > > information around an opaque UUID. > > > > > > > You are actually arguing against your own design here, unfortunately. If you > > look at your design carefully, it is your design actually require to have a > > 3rd > > party code to figure out the VM and virtual gpu device relation as it is > > never documented in the sysfs. > > No. It's not about figuring out the relation between VM and vGPU. It's about > figuring out some raw information about vGPU itself, such as: > > vgpu0@:00:02.0, I can immediately know it's the 1st instance created on > :00:02.0 device. While with an UUID, it doesn't speak anything useful > here. Right, I get your point, but this requires the qemu to know this virtual device is created on that physical device, isn't it, given you already know that the virtual device is created on the physical device :00:02.0, right? > > This is a typical way how device nodes are created within sysfs, e.g. on my > platform: > > $ ls
Re: [Qemu-devel] [RFC PATCH v1 1/1] vGPU core driver : to provide common interface for vGPU.
Hi, > The answer is simple, having a UUID as part of the device name will give you a > unique sysfs path that will be opened by QEMU. A descriptive name will work too, and I think it'll be easier to make those names persistent because you don't have to store the uuids somewhere to re-create the same setup afer reboot. > If you are worried about losing meaningful name here, we can create a sysfs > file > to capture the vendor device description if you like. You can also store the uuid in a sysfs file ... cheers, Gerd
Re: [Qemu-devel] [RFC PATCH v1 1/1] vGPU core driver : to provide common interface for vGPU.
> From: Neo Jia > Sent: Wednesday, February 17, 2016 1:38 PM > > > > > > > > > > > > > > Hi Kevin, > > > > > > The answer is simple, having a UUID as part of the device name will give > > > you a > > > unique sysfs path that will be opened by QEMU. > > > > > > vgpu-vendor-0 and vgpu-vendor-1 will not be unique as we can have multiple > > > virtual gpu devices per VM coming from same or different physical devices. > > > > That is not a problem. We can add physical device info too like > > vgpu-vendor-0-0, > > vgpu-vendor-1-0, ... > > > > Please note Qemu doesn't care about the actual name. It just accepts a > > sysfs path > > to open. > > Hi Kevin, > > No, I think you are making things even more complicated than it is required, > also it is not generic anymore as you are requiring the QEMU to know more than > he needs to. > > The way you name those devices will require QEMU to know the relation > between virtual devices and physical devices. I don't think that is good. I don't think you get my point. Look at how device is assigned in Qemu today: -device vfio-pci,host=02:00.0 Then in a recent patch from Alex, Qemu will accept sysfsdev as well: -device vfio-pci,sysfsdev=/sys/devices/pci:00/:00:1c.0/:02:00.0 Then with vgu (one example from Alex's original post): -device vfio-pci,sysfsdev=/sys/devices/virtual/intel-vgpu/vgpu0@:00:02.0 Qemu doesn't need to know the relation between virtual/physical devices at all. It's just a path regardless of how vgpu name is created (either with your UUID proposal or my descriptive string proposal) > > My flow is like this: > > libvirt creats a VM object, it will have a UUID. then it will use the UUID to > create virtual gpu devices, then it will pass the UUID to the QEMU (actually > QEMU already has the VM UUID), then it will just open up the unique path. If you look at above example, it's not UUID itself being passed to Qemu. It's the sysfsdev path. > > Also, you need to consider those 0-0 numbers are not generic as the UUID. that encoding could be flexible to include any meaningful string. libvirt can itself manages how UUID is mapped to an actual vgpu name. > > > > > > > > If you are worried about losing meaningful name here, we can create a > > > sysfs file > > > to capture the vendor device description if you like. > > > > > > > Having the vgpu name descriptive is more informative imo. User can simply > > check > > sysfs names to know raw information w/o relying on 3rd party agent to query > > information around an opaque UUID. > > > > You are actually arguing against your own design here, unfortunately. If you > look at your design carefully, it is your design actually require to have a > 3rd > party code to figure out the VM and virtual gpu device relation as it is > never documented in the sysfs. No. It's not about figuring out the relation between VM and vGPU. It's about figuring out some raw information about vGPU itself, such as: vgpu0@:00:02.0, I can immediately know it's the 1st instance created on :00:02.0 device. While with an UUID, it doesn't speak anything useful here. This is a typical way how device nodes are created within sysfs, e.g. on my platform: $ ls /sys/class/drm/ card0/ card0-DP-2/ card0-HDMI-A-2/ controlD64/ card0-DP-1/ card0-HDMI-A-1/ card0-VGA-1/version $ ls /sys/bus/pci/devices/ :00:00.0 :00:14.0 :00:1a.0 :00:1c.1 :00:1f.2 :00:02.0 :00:16.0 :00:1b.0 :00:1d.0 :00:1f.3 :00:03.0 :00:19.0 :00:1c.0 :00:1f.0 :02:00.0 We'd better keep such style when creating vgpu nodes in sysfs. UUID is at most anther info suffixed to the default string (or in another file), if necessary. > > In our current design, it doesn't require any 3rd party agent as the VM UUID > is > part of the QEMU command already, and the VM UUID is already embedded within > the > virtual device path. > > Also, it doesn't require 3rd party to retrieve information as the virtual > device > will just be a directory, we will have another file within each virtual gpu > device directory, you can always cat the file to retrieve vendor information. > > Let's use the UUID-$vgpu_idx as the virtual device directory name plus a > vendor > description file within that directory, so we don't lose any additional > information, also capture the VM and virtual device relation. > I'm fine to have another node to provide more vendor specific information. But I don't want to make UUID mandatory when creating a vGPU instance, as explained above. Today we can create VMs in KVM w/o using libvirt, and w/o the need of allocating any UUID. Same thing should be supported for vgpu feature too. Thanks Kevin
Re: [Qemu-devel] [RFC PATCH v1 1/1] vGPU core driver : to provide common interface for vGPU.
On Tue, Feb 16, 2016 at 10:09:43PM -0700, Eric Blake wrote: > * PGP Signed by an unknown key > > On 02/16/2016 10:04 PM, Tian, Kevin wrote: > > > ...rather than making readers scroll through 16k bytes of repetitions of > the same things they saw earlier in the thread, but getting worse with > each iteration due to excessive quoting. > Hi Eric, Sorry about that, I will pay attention to this. Thanks, Neo > -- > Eric Blake eblake redhat com+1-919-301-3266 > Libvirt virtualization library http://libvirt.org > > > * Unknown Key > * 0x2527436A
Re: [Qemu-devel] [RFC PATCH v1 1/1] vGPU core driver : to provide common interface for vGPU.
On Wed, Feb 17, 2016 at 05:04:31AM +, Tian, Kevin wrote:
> > From: Neo Jia
> > Sent: Wednesday, February 17, 2016 12:18 PM
> >
> > On Wed, Feb 17, 2016 at 03:31:24AM +, Tian, Kevin wrote:
> > > > From: Neo Jia [mailto:c...@nvidia.com]
> > > > Sent: Tuesday, February 16, 2016 4:49 PM
> > > >
> > > > On Tue, Feb 16, 2016 at 08:10:42AM +, Tian, Kevin wrote:
> > > > > > From: Neo Jia [mailto:c...@nvidia.com]
> > > > > > Sent: Tuesday, February 16, 2016 3:53 PM
> > > > > >
> > > > > > On Tue, Feb 16, 2016 at 07:40:47AM +, Tian, Kevin wrote:
> > > > > > > > From: Neo Jia [mailto:c...@nvidia.com]
> > > > > > > > Sent: Tuesday, February 16, 2016 3:37 PM
> > > > > > > >
> > > > > > > > On Tue, Feb 16, 2016 at 07:27:09AM +, Tian, Kevin wrote:
> > > > > > > > > > From: Neo Jia [mailto:c...@nvidia.com]
> > > > > > > > > > Sent: Tuesday, February 16, 2016 3:13 PM
> > > > > > > > > >
> > > > > > > > > > On Tue, Feb 16, 2016 at 06:49:30AM +, Tian, Kevin wrote:
> > > > > > > > > > > > From: Alex Williamson
> > > > > > > > > > > > [mailto:alex.william...@redhat.com]
> > > > > > > > > > > > Sent: Thursday, February 04, 2016 3:33 AM
> > > > > > > > > > > >
> > > > > > > > > > > > On Wed, 2016-02-03 at 09:28 +0100, Gerd Hoffmann wrote:
> > > > > > > > > > > > > Hi,
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Actually I have a long puzzle in this area.
> > > > > > > > > > > > > > Definitely libvirt will
> > use
> > > > UUID
> > > > > > to
> > > > > > > > > > > > > > mark a VM. And obviously UUID is not recorded
> > > > > > > > > > > > > > within KVM.
> > Then
> > > > how
> > > > > > does
> > > > > > > > > > > > > > libvirt talk to KVM based on UUID? It could be a
> > > > > > > > > > > > > > good reference
> > to
> > > > this
> > > > > > design.
> > > > > > > > > > > > >
> > > > > > > > > > > > > libvirt keeps track which qemu instance belongs to
> > > > > > > > > > > > > which vm.
> > > > > > > > > > > > > qemu also gets started with "-uuid ...", so one can
> > > > > > > > > > > > > query qemu
> > via
> > > > > > > > > > > > > monitor ("info uuid") to figure what the uuid is. It
> > > > > > > > > > > > > is also in the
> > > > > > > > > > > > > smbios tables so the guest can see it in the system
> > > > > > > > > > > > > information
> > table.
> > > > > > > > > > > > >
> > > > > > > > > > > > > The uuid is not visible to the kernel though, the kvm
> > > > > > > > > > > > > kernel driver
> > > > > > > > > > > > > doesn't know what the uuid is (and neither does
> > > > > > > > > > > > > vfio). qemu uses
> > > > file
> > > > > > > > > > > > > handles to talk to both kvm and vfio. qemu notifies
> > > > > > > > > > > > > both kvm
> > and
> > > > vfio
> > > > > > > > > > > > > about anything relevant events (guest address space
> > > > > > > > > > > > > changes
> > etc)
> > > > and
> > > > > > > > > > > > > connects file descriptors (eventfd -> irqfd).
> > > > > > > > > > > >
> > > > > > > > > > > > I think the original link to using a VM UUID for the
> > > > > > > > > > > > vGPU comes from
> > > > > > > > > > > > NVIDIA having a userspace component which might get
> > > > > > > > > > > > launched
> > from
> > > > a udev
> > > > > > > > > > > > event as the vGPU is created or the set of vGPUs within
> > > > > > > > > > > > that UUID
> > is
> > > > > > > > > > > > started. Using the VM UUID then gives them a way to
> > > > > > > > > > > > associate
> > that
> > > > > > > > > > > > userspace process with a VM instance. Maybe it could
> > > > > > > > > > > > register with
> > > > > > > > > > > > libvirt for some sort of service provided for the VM, I
> > > > > > > > > > > > don't know.
> > > > > > > > > > >
> > > > > > > > > > > Intel doesn't have this requirement. It should be enough
> > > > > > > > > > > as long as
> > > > > > > > > > > libvirt maintains which sysfs vgpu node is associated to
> > > > > > > > > > > a VM UUID.
> > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > > qemu needs a sysfs node as handle to the vfio device,
> > > > > > > > > > > > > something
> > > > > > > > > > > > > like /sys/devices/virtual/vgpu/. can be
> > > > > > > > > > > > > a uuid
> > if
> > > > you
> > > > > > want
> > > > > > > > > > > > > have it that way, but it could be pretty much
> > > > > > > > > > > > > anything. The sysfs
> > node
> > > > > > > > > > > > > will probably show up as-is in the libvirt xml when
> > > > > > > > > > > > > assign a vgpu
> > to
> > > > a
> > > > > > > > > > > > > vm. So the name should be something stable (i.e.
> > > > > > > > > > > > > when using
> > a uuid
> > > > as
> > > > > > > > > > > > > name you should better not generate a new one on each
> > > > > > > > > > > > > boot).
> > > > > > > > > > > >
> > > > > > > > > > > > Actually I don't think there's really a persistent
> > > > > > > > > > > > naming issue, that's
> > > > > > > > > > > > probably where we diverge from the SR-IOV model.
> > > > > > > > > > > > SR-IOV
Re: [Qemu-devel] [RFC PATCH v1 1/1] vGPU core driver : to provide common interface for vGPU.
On 02/16/2016 10:04 PM, Tian, Kevin wrote: >> From: Neo Jia [meta-comment] ... On Wed, 2016-02-03 at 09:28 +0100, Gerd Hoffmann wrote: > Hi, > >> Actually I have a long puzzle in this area. Definitely libvirt >> will >> use UUID >> to With this much quoting, your mailer is breaking down. It's not only okay, but encouraged, to trim your message to just what you are directly replying to... >> Hi Kevin, >> >> The answer is simple, having a UUID as part of the device name will give you >> a >> unique sysfs path that will be opened by QEMU. >> >> vgpu-vendor-0 and vgpu-vendor-1 will not be unique as we can have multiple >> virtual gpu devices per VM coming from same or different physical devices. > > That is not a problem. We can add physical device info too like > vgpu-vendor-0-0, > vgpu-vendor-1-0, ... ...rather than making readers scroll through 16k bytes of repetitions of the same things they saw earlier in the thread, but getting worse with each iteration due to excessive quoting. -- Eric Blake eblake redhat com+1-919-301-3266 Libvirt virtualization library http://libvirt.org signature.asc Description: OpenPGP digital signature
Re: [Qemu-devel] [RFC PATCH v1 1/1] vGPU core driver : to provide common interface for vGPU.
> From: Neo Jia
> Sent: Wednesday, February 17, 2016 12:18 PM
>
> On Wed, Feb 17, 2016 at 03:31:24AM +, Tian, Kevin wrote:
> > > From: Neo Jia [mailto:c...@nvidia.com]
> > > Sent: Tuesday, February 16, 2016 4:49 PM
> > >
> > > On Tue, Feb 16, 2016 at 08:10:42AM +, Tian, Kevin wrote:
> > > > > From: Neo Jia [mailto:c...@nvidia.com]
> > > > > Sent: Tuesday, February 16, 2016 3:53 PM
> > > > >
> > > > > On Tue, Feb 16, 2016 at 07:40:47AM +, Tian, Kevin wrote:
> > > > > > > From: Neo Jia [mailto:c...@nvidia.com]
> > > > > > > Sent: Tuesday, February 16, 2016 3:37 PM
> > > > > > >
> > > > > > > On Tue, Feb 16, 2016 at 07:27:09AM +, Tian, Kevin wrote:
> > > > > > > > > From: Neo Jia [mailto:c...@nvidia.com]
> > > > > > > > > Sent: Tuesday, February 16, 2016 3:13 PM
> > > > > > > > >
> > > > > > > > > On Tue, Feb 16, 2016 at 06:49:30AM +, Tian, Kevin wrote:
> > > > > > > > > > > From: Alex Williamson [mailto:alex.william...@redhat.com]
> > > > > > > > > > > Sent: Thursday, February 04, 2016 3:33 AM
> > > > > > > > > > >
> > > > > > > > > > > On Wed, 2016-02-03 at 09:28 +0100, Gerd Hoffmann wrote:
> > > > > > > > > > > > Hi,
> > > > > > > > > > > >
> > > > > > > > > > > > > Actually I have a long puzzle in this area.
> > > > > > > > > > > > > Definitely libvirt will
> use
> > > UUID
> > > > > to
> > > > > > > > > > > > > mark a VM. And obviously UUID is not recorded within
> > > > > > > > > > > > > KVM.
> Then
> > > how
> > > > > does
> > > > > > > > > > > > > libvirt talk to KVM based on UUID? It could be a good
> > > > > > > > > > > > > reference
> to
> > > this
> > > > > design.
> > > > > > > > > > > >
> > > > > > > > > > > > libvirt keeps track which qemu instance belongs to
> > > > > > > > > > > > which vm.
> > > > > > > > > > > > qemu also gets started with "-uuid ...", so one can
> > > > > > > > > > > > query qemu
> via
> > > > > > > > > > > > monitor ("info uuid") to figure what the uuid is. It
> > > > > > > > > > > > is also in the
> > > > > > > > > > > > smbios tables so the guest can see it in the system
> > > > > > > > > > > > information
> table.
> > > > > > > > > > > >
> > > > > > > > > > > > The uuid is not visible to the kernel though, the kvm
> > > > > > > > > > > > kernel driver
> > > > > > > > > > > > doesn't know what the uuid is (and neither does vfio).
> > > > > > > > > > > > qemu uses
> > > file
> > > > > > > > > > > > handles to talk to both kvm and vfio. qemu notifies
> > > > > > > > > > > > both kvm
> and
> > > vfio
> > > > > > > > > > > > about anything relevant events (guest address space
> > > > > > > > > > > > changes
> etc)
> > > and
> > > > > > > > > > > > connects file descriptors (eventfd -> irqfd).
> > > > > > > > > > >
> > > > > > > > > > > I think the original link to using a VM UUID for the vGPU
> > > > > > > > > > > comes from
> > > > > > > > > > > NVIDIA having a userspace component which might get
> > > > > > > > > > > launched
> from
> > > a udev
> > > > > > > > > > > event as the vGPU is created or the set of vGPUs within
> > > > > > > > > > > that UUID
> is
> > > > > > > > > > > started. Using the VM UUID then gives them a way to
> > > > > > > > > > > associate
> that
> > > > > > > > > > > userspace process with a VM instance. Maybe it could
> > > > > > > > > > > register with
> > > > > > > > > > > libvirt for some sort of service provided for the VM, I
> > > > > > > > > > > don't know.
> > > > > > > > > >
> > > > > > > > > > Intel doesn't have this requirement. It should be enough as
> > > > > > > > > > long as
> > > > > > > > > > libvirt maintains which sysfs vgpu node is associated to a
> > > > > > > > > > VM UUID.
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > > qemu needs a sysfs node as handle to the vfio device,
> > > > > > > > > > > > something
> > > > > > > > > > > > like /sys/devices/virtual/vgpu/. can be a
> > > > > > > > > > > > uuid
> if
> > > you
> > > > > want
> > > > > > > > > > > > have it that way, but it could be pretty much anything.
> > > > > > > > > > > > The sysfs
> node
> > > > > > > > > > > > will probably show up as-is in the libvirt xml when
> > > > > > > > > > > > assign a vgpu
> to
> > > a
> > > > > > > > > > > > vm. So the name should be something stable (i.e. when
> > > > > > > > > > > > using
> a uuid
> > > as
> > > > > > > > > > > > name you should better not generate a new one on each
> > > > > > > > > > > > boot).
> > > > > > > > > > >
> > > > > > > > > > > Actually I don't think there's really a persistent naming
> > > > > > > > > > > issue, that's
> > > > > > > > > > > probably where we diverge from the SR-IOV model. SR-IOV
> > > > > > > > > > > cannot
> > > > > > > > > > > dynamically add a new VF, it needs to reset the number of
> > > > > > > > > > > VFs to
> zero,
> > > > > > > > > > > then re-allocate all of them up to the new desired count.
> > > > > > > > > > > That has
> some
> > > > > > > > > > > obvious implications. I think with both vendors
Re: [Qemu-devel] [RFC PATCH v1 1/1] vGPU core driver : to provide common interface for vGPU.
On Wed, Feb 17, 2016 at 03:31:24AM +, Tian, Kevin wrote:
> > From: Neo Jia [mailto:c...@nvidia.com]
> > Sent: Tuesday, February 16, 2016 4:49 PM
> >
> > On Tue, Feb 16, 2016 at 08:10:42AM +, Tian, Kevin wrote:
> > > > From: Neo Jia [mailto:c...@nvidia.com]
> > > > Sent: Tuesday, February 16, 2016 3:53 PM
> > > >
> > > > On Tue, Feb 16, 2016 at 07:40:47AM +, Tian, Kevin wrote:
> > > > > > From: Neo Jia [mailto:c...@nvidia.com]
> > > > > > Sent: Tuesday, February 16, 2016 3:37 PM
> > > > > >
> > > > > > On Tue, Feb 16, 2016 at 07:27:09AM +, Tian, Kevin wrote:
> > > > > > > > From: Neo Jia [mailto:c...@nvidia.com]
> > > > > > > > Sent: Tuesday, February 16, 2016 3:13 PM
> > > > > > > >
> > > > > > > > On Tue, Feb 16, 2016 at 06:49:30AM +, Tian, Kevin wrote:
> > > > > > > > > > From: Alex Williamson [mailto:alex.william...@redhat.com]
> > > > > > > > > > Sent: Thursday, February 04, 2016 3:33 AM
> > > > > > > > > >
> > > > > > > > > > On Wed, 2016-02-03 at 09:28 +0100, Gerd Hoffmann wrote:
> > > > > > > > > > > Hi,
> > > > > > > > > > >
> > > > > > > > > > > > Actually I have a long puzzle in this area. Definitely
> > > > > > > > > > > > libvirt will use
> > UUID
> > > > to
> > > > > > > > > > > > mark a VM. And obviously UUID is not recorded within
> > > > > > > > > > > > KVM. Then
> > how
> > > > does
> > > > > > > > > > > > libvirt talk to KVM based on UUID? It could be a good
> > > > > > > > > > > > reference to
> > this
> > > > design.
> > > > > > > > > > >
> > > > > > > > > > > libvirt keeps track which qemu instance belongs to which
> > > > > > > > > > > vm.
> > > > > > > > > > > qemu also gets started with "-uuid ...", so one can query
> > > > > > > > > > > qemu via
> > > > > > > > > > > monitor ("info uuid") to figure what the uuid is. It is
> > > > > > > > > > > also in the
> > > > > > > > > > > smbios tables so the guest can see it in the system
> > > > > > > > > > > information table.
> > > > > > > > > > >
> > > > > > > > > > > The uuid is not visible to the kernel though, the kvm
> > > > > > > > > > > kernel driver
> > > > > > > > > > > doesn't know what the uuid is (and neither does vfio).
> > > > > > > > > > > qemu uses
> > file
> > > > > > > > > > > handles to talk to both kvm and vfio. qemu notifies both
> > > > > > > > > > > kvm and
> > vfio
> > > > > > > > > > > about anything relevant events (guest address space
> > > > > > > > > > > changes etc)
> > and
> > > > > > > > > > > connects file descriptors (eventfd -> irqfd).
> > > > > > > > > >
> > > > > > > > > > I think the original link to using a VM UUID for the vGPU
> > > > > > > > > > comes from
> > > > > > > > > > NVIDIA having a userspace component which might get
> > > > > > > > > > launched from
> > a udev
> > > > > > > > > > event as the vGPU is created or the set of vGPUs within
> > > > > > > > > > that UUID is
> > > > > > > > > > started. Using the VM UUID then gives them a way to
> > > > > > > > > > associate that
> > > > > > > > > > userspace process with a VM instance. Maybe it could
> > > > > > > > > > register with
> > > > > > > > > > libvirt for some sort of service provided for the VM, I
> > > > > > > > > > don't know.
> > > > > > > > >
> > > > > > > > > Intel doesn't have this requirement. It should be enough as
> > > > > > > > > long as
> > > > > > > > > libvirt maintains which sysfs vgpu node is associated to a VM
> > > > > > > > > UUID.
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > qemu needs a sysfs node as handle to the vfio device,
> > > > > > > > > > > something
> > > > > > > > > > > like /sys/devices/virtual/vgpu/. can be a
> > > > > > > > > > > uuid if
> > you
> > > > want
> > > > > > > > > > > have it that way, but it could be pretty much anything.
> > > > > > > > > > > The sysfs node
> > > > > > > > > > > will probably show up as-is in the libvirt xml when
> > > > > > > > > > > assign a vgpu to
> > a
> > > > > > > > > > > vm. So the name should be something stable (i.e. when
> > > > > > > > > > > using a uuid
> > as
> > > > > > > > > > > name you should better not generate a new one on each
> > > > > > > > > > > boot).
> > > > > > > > > >
> > > > > > > > > > Actually I don't think there's really a persistent naming
> > > > > > > > > > issue, that's
> > > > > > > > > > probably where we diverge from the SR-IOV model. SR-IOV
> > > > > > > > > > cannot
> > > > > > > > > > dynamically add a new VF, it needs to reset the number of
> > > > > > > > > > VFs to zero,
> > > > > > > > > > then re-allocate all of them up to the new desired count.
> > > > > > > > > > That has some
> > > > > > > > > > obvious implications. I think with both vendors here, we
> > > > > > > > > > can
> > > > > > > > > > dynamically allocate new vGPUs, so I would expect that
> > > > > > > > > > libvirt would
> > > > > > > > > > create each vGPU instance as it's needed. None would be
> > > > > > > > > > created by
> > > > > > > > > > default without user interaction.
Re: [Qemu-devel] [RFC PATCH v1 1/1] vGPU core driver : to provide common interface for vGPU.
> From: Neo Jia [mailto:c...@nvidia.com]
> Sent: Tuesday, February 16, 2016 4:49 PM
>
> On Tue, Feb 16, 2016 at 08:10:42AM +, Tian, Kevin wrote:
> > > From: Neo Jia [mailto:c...@nvidia.com]
> > > Sent: Tuesday, February 16, 2016 3:53 PM
> > >
> > > On Tue, Feb 16, 2016 at 07:40:47AM +, Tian, Kevin wrote:
> > > > > From: Neo Jia [mailto:c...@nvidia.com]
> > > > > Sent: Tuesday, February 16, 2016 3:37 PM
> > > > >
> > > > > On Tue, Feb 16, 2016 at 07:27:09AM +, Tian, Kevin wrote:
> > > > > > > From: Neo Jia [mailto:c...@nvidia.com]
> > > > > > > Sent: Tuesday, February 16, 2016 3:13 PM
> > > > > > >
> > > > > > > On Tue, Feb 16, 2016 at 06:49:30AM +, Tian, Kevin wrote:
> > > > > > > > > From: Alex Williamson [mailto:alex.william...@redhat.com]
> > > > > > > > > Sent: Thursday, February 04, 2016 3:33 AM
> > > > > > > > >
> > > > > > > > > On Wed, 2016-02-03 at 09:28 +0100, Gerd Hoffmann wrote:
> > > > > > > > > > Hi,
> > > > > > > > > >
> > > > > > > > > > > Actually I have a long puzzle in this area. Definitely
> > > > > > > > > > > libvirt will use
> UUID
> > > to
> > > > > > > > > > > mark a VM. And obviously UUID is not recorded within KVM.
> > > > > > > > > > > Then
> how
> > > does
> > > > > > > > > > > libvirt talk to KVM based on UUID? It could be a good
> > > > > > > > > > > reference to
> this
> > > design.
> > > > > > > > > >
> > > > > > > > > > libvirt keeps track which qemu instance belongs to which vm.
> > > > > > > > > > qemu also gets started with "-uuid ...", so one can query
> > > > > > > > > > qemu via
> > > > > > > > > > monitor ("info uuid") to figure what the uuid is. It is
> > > > > > > > > > also in the
> > > > > > > > > > smbios tables so the guest can see it in the system
> > > > > > > > > > information table.
> > > > > > > > > >
> > > > > > > > > > The uuid is not visible to the kernel though, the kvm
> > > > > > > > > > kernel driver
> > > > > > > > > > doesn't know what the uuid is (and neither does vfio).
> > > > > > > > > > qemu uses
> file
> > > > > > > > > > handles to talk to both kvm and vfio. qemu notifies both
> > > > > > > > > > kvm and
> vfio
> > > > > > > > > > about anything relevant events (guest address space changes
> > > > > > > > > > etc)
> and
> > > > > > > > > > connects file descriptors (eventfd -> irqfd).
> > > > > > > > >
> > > > > > > > > I think the original link to using a VM UUID for the vGPU
> > > > > > > > > comes from
> > > > > > > > > NVIDIA having a userspace component which might get launched
> > > > > > > > > from
> a udev
> > > > > > > > > event as the vGPU is created or the set of vGPUs within that
> > > > > > > > > UUID is
> > > > > > > > > started. Using the VM UUID then gives them a way to
> > > > > > > > > associate that
> > > > > > > > > userspace process with a VM instance. Maybe it could
> > > > > > > > > register with
> > > > > > > > > libvirt for some sort of service provided for the VM, I don't
> > > > > > > > > know.
> > > > > > > >
> > > > > > > > Intel doesn't have this requirement. It should be enough as
> > > > > > > > long as
> > > > > > > > libvirt maintains which sysfs vgpu node is associated to a VM
> > > > > > > > UUID.
> > > > > > > >
> > > > > > > > >
> > > > > > > > > > qemu needs a sysfs node as handle to the vfio device,
> > > > > > > > > > something
> > > > > > > > > > like /sys/devices/virtual/vgpu/. can be a
> > > > > > > > > > uuid if
> you
> > > want
> > > > > > > > > > have it that way, but it could be pretty much anything.
> > > > > > > > > > The sysfs node
> > > > > > > > > > will probably show up as-is in the libvirt xml when assign
> > > > > > > > > > a vgpu to
> a
> > > > > > > > > > vm. So the name should be something stable (i.e. when
> > > > > > > > > > using a uuid
> as
> > > > > > > > > > name you should better not generate a new one on each boot).
> > > > > > > > >
> > > > > > > > > Actually I don't think there's really a persistent naming
> > > > > > > > > issue, that's
> > > > > > > > > probably where we diverge from the SR-IOV model. SR-IOV
> > > > > > > > > cannot
> > > > > > > > > dynamically add a new VF, it needs to reset the number of VFs
> > > > > > > > > to zero,
> > > > > > > > > then re-allocate all of them up to the new desired count.
> > > > > > > > > That has some
> > > > > > > > > obvious implications. I think with both vendors here, we can
> > > > > > > > > dynamically allocate new vGPUs, so I would expect that
> > > > > > > > > libvirt would
> > > > > > > > > create each vGPU instance as it's needed. None would be
> > > > > > > > > created by
> > > > > > > > > default without user interaction.
> > > > > > > > >
> > > > > > > > > Personally I think using a UUID makes sense, but it needs to
> > > > > > > > > be
> > > > > > > > > userspace policy whether that UUID has any implicit meaning
> > > > > > > > > like
> > > > > > > > > matching the VM UUID. Having an index within a UUID bothers
> > > > > > > > > me a
> bit,
> > > >
Re: [Qemu-devel] [RFC PATCH v1 1/1] vGPU core driver : to provide common interface for vGPU.
On Tue, Feb 16, 2016 at 08:10:42AM +, Tian, Kevin wrote:
> > From: Neo Jia [mailto:c...@nvidia.com]
> > Sent: Tuesday, February 16, 2016 3:53 PM
> >
> > On Tue, Feb 16, 2016 at 07:40:47AM +, Tian, Kevin wrote:
> > > > From: Neo Jia [mailto:c...@nvidia.com]
> > > > Sent: Tuesday, February 16, 2016 3:37 PM
> > > >
> > > > On Tue, Feb 16, 2016 at 07:27:09AM +, Tian, Kevin wrote:
> > > > > > From: Neo Jia [mailto:c...@nvidia.com]
> > > > > > Sent: Tuesday, February 16, 2016 3:13 PM
> > > > > >
> > > > > > On Tue, Feb 16, 2016 at 06:49:30AM +, Tian, Kevin wrote:
> > > > > > > > From: Alex Williamson [mailto:alex.william...@redhat.com]
> > > > > > > > Sent: Thursday, February 04, 2016 3:33 AM
> > > > > > > >
> > > > > > > > On Wed, 2016-02-03 at 09:28 +0100, Gerd Hoffmann wrote:
> > > > > > > > > Hi,
> > > > > > > > >
> > > > > > > > > > Actually I have a long puzzle in this area. Definitely
> > > > > > > > > > libvirt will use UUID
> > to
> > > > > > > > > > mark a VM. And obviously UUID is not recorded within KVM.
> > > > > > > > > > Then how
> > does
> > > > > > > > > > libvirt talk to KVM based on UUID? It could be a good
> > > > > > > > > > reference to this
> > design.
> > > > > > > > >
> > > > > > > > > libvirt keeps track which qemu instance belongs to which vm.
> > > > > > > > > qemu also gets started with "-uuid ...", so one can query
> > > > > > > > > qemu via
> > > > > > > > > monitor ("info uuid") to figure what the uuid is. It is also
> > > > > > > > > in the
> > > > > > > > > smbios tables so the guest can see it in the system
> > > > > > > > > information table.
> > > > > > > > >
> > > > > > > > > The uuid is not visible to the kernel though, the kvm kernel
> > > > > > > > > driver
> > > > > > > > > doesn't know what the uuid is (and neither does vfio). qemu
> > > > > > > > > uses file
> > > > > > > > > handles to talk to both kvm and vfio. qemu notifies both kvm
> > > > > > > > > and vfio
> > > > > > > > > about anything relevant events (guest address space changes
> > > > > > > > > etc) and
> > > > > > > > > connects file descriptors (eventfd -> irqfd).
> > > > > > > >
> > > > > > > > I think the original link to using a VM UUID for the vGPU comes
> > > > > > > > from
> > > > > > > > NVIDIA having a userspace component which might get launched
> > > > > > > > from a udev
> > > > > > > > event as the vGPU is created or the set of vGPUs within that
> > > > > > > > UUID is
> > > > > > > > started. Using the VM UUID then gives them a way to associate
> > > > > > > > that
> > > > > > > > userspace process with a VM instance. Maybe it could register
> > > > > > > > with
> > > > > > > > libvirt for some sort of service provided for the VM, I don't
> > > > > > > > know.
> > > > > > >
> > > > > > > Intel doesn't have this requirement. It should be enough as long
> > > > > > > as
> > > > > > > libvirt maintains which sysfs vgpu node is associated to a VM
> > > > > > > UUID.
> > > > > > >
> > > > > > > >
> > > > > > > > > qemu needs a sysfs node as handle to the vfio device,
> > > > > > > > > something
> > > > > > > > > like /sys/devices/virtual/vgpu/. can be a uuid
> > > > > > > > > if you
> > want
> > > > > > > > > have it that way, but it could be pretty much anything. The
> > > > > > > > > sysfs node
> > > > > > > > > will probably show up as-is in the libvirt xml when assign a
> > > > > > > > > vgpu to a
> > > > > > > > > vm. So the name should be something stable (i.e. when using
> > > > > > > > > a uuid as
> > > > > > > > > name you should better not generate a new one on each boot).
> > > > > > > >
> > > > > > > > Actually I don't think there's really a persistent naming
> > > > > > > > issue, that's
> > > > > > > > probably where we diverge from the SR-IOV model. SR-IOV cannot
> > > > > > > > dynamically add a new VF, it needs to reset the number of VFs
> > > > > > > > to zero,
> > > > > > > > then re-allocate all of them up to the new desired count. That
> > > > > > > > has some
> > > > > > > > obvious implications. I think with both vendors here, we can
> > > > > > > > dynamically allocate new vGPUs, so I would expect that libvirt
> > > > > > > > would
> > > > > > > > create each vGPU instance as it's needed. None would be
> > > > > > > > created by
> > > > > > > > default without user interaction.
> > > > > > > >
> > > > > > > > Personally I think using a UUID makes sense, but it needs to be
> > > > > > > > userspace policy whether that UUID has any implicit meaning like
> > > > > > > > matching the VM UUID. Having an index within a UUID bothers me
> > > > > > > > a bit,
> > > > > > > > but it doesn't seem like too much of a concession to enable the
> > > > > > > > use case
> > > > > > > > that NVIDIA is trying to achieve. Thanks,
> > > > > > > >
> > > > > > >
> > > > > > > I would prefer to making UUID an optional parameter, while not
> > > > > > > tieing
> > > > > > > sysfs vgpu naming to UUID. This would be more flexible to
Re: [Qemu-devel] [RFC PATCH v1 1/1] vGPU core driver : to provide common interface for vGPU.
> From: Neo Jia [mailto:c...@nvidia.com]
> Sent: Tuesday, February 16, 2016 3:53 PM
>
> On Tue, Feb 16, 2016 at 07:40:47AM +, Tian, Kevin wrote:
> > > From: Neo Jia [mailto:c...@nvidia.com]
> > > Sent: Tuesday, February 16, 2016 3:37 PM
> > >
> > > On Tue, Feb 16, 2016 at 07:27:09AM +, Tian, Kevin wrote:
> > > > > From: Neo Jia [mailto:c...@nvidia.com]
> > > > > Sent: Tuesday, February 16, 2016 3:13 PM
> > > > >
> > > > > On Tue, Feb 16, 2016 at 06:49:30AM +, Tian, Kevin wrote:
> > > > > > > From: Alex Williamson [mailto:alex.william...@redhat.com]
> > > > > > > Sent: Thursday, February 04, 2016 3:33 AM
> > > > > > >
> > > > > > > On Wed, 2016-02-03 at 09:28 +0100, Gerd Hoffmann wrote:
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > > > Actually I have a long puzzle in this area. Definitely
> > > > > > > > > libvirt will use UUID
> to
> > > > > > > > > mark a VM. And obviously UUID is not recorded within KVM.
> > > > > > > > > Then how
> does
> > > > > > > > > libvirt talk to KVM based on UUID? It could be a good
> > > > > > > > > reference to this
> design.
> > > > > > > >
> > > > > > > > libvirt keeps track which qemu instance belongs to which vm.
> > > > > > > > qemu also gets started with "-uuid ...", so one can query qemu
> > > > > > > > via
> > > > > > > > monitor ("info uuid") to figure what the uuid is. It is also
> > > > > > > > in the
> > > > > > > > smbios tables so the guest can see it in the system information
> > > > > > > > table.
> > > > > > > >
> > > > > > > > The uuid is not visible to the kernel though, the kvm kernel
> > > > > > > > driver
> > > > > > > > doesn't know what the uuid is (and neither does vfio). qemu
> > > > > > > > uses file
> > > > > > > > handles to talk to both kvm and vfio. qemu notifies both kvm
> > > > > > > > and vfio
> > > > > > > > about anything relevant events (guest address space changes
> > > > > > > > etc) and
> > > > > > > > connects file descriptors (eventfd -> irqfd).
> > > > > > >
> > > > > > > I think the original link to using a VM UUID for the vGPU comes
> > > > > > > from
> > > > > > > NVIDIA having a userspace component which might get launched from
> > > > > > > a udev
> > > > > > > event as the vGPU is created or the set of vGPUs within that UUID
> > > > > > > is
> > > > > > > started. Using the VM UUID then gives them a way to associate
> > > > > > > that
> > > > > > > userspace process with a VM instance. Maybe it could register
> > > > > > > with
> > > > > > > libvirt for some sort of service provided for the VM, I don't
> > > > > > > know.
> > > > > >
> > > > > > Intel doesn't have this requirement. It should be enough as long as
> > > > > > libvirt maintains which sysfs vgpu node is associated to a VM UUID.
> > > > > >
> > > > > > >
> > > > > > > > qemu needs a sysfs node as handle to the vfio device, something
> > > > > > > > like /sys/devices/virtual/vgpu/. can be a uuid if
> > > > > > > > you
> want
> > > > > > > > have it that way, but it could be pretty much anything. The
> > > > > > > > sysfs node
> > > > > > > > will probably show up as-is in the libvirt xml when assign a
> > > > > > > > vgpu to a
> > > > > > > > vm. So the name should be something stable (i.e. when using a
> > > > > > > > uuid as
> > > > > > > > name you should better not generate a new one on each boot).
> > > > > > >
> > > > > > > Actually I don't think there's really a persistent naming issue,
> > > > > > > that's
> > > > > > > probably where we diverge from the SR-IOV model. SR-IOV cannot
> > > > > > > dynamically add a new VF, it needs to reset the number of VFs to
> > > > > > > zero,
> > > > > > > then re-allocate all of them up to the new desired count. That
> > > > > > > has some
> > > > > > > obvious implications. I think with both vendors here, we can
> > > > > > > dynamically allocate new vGPUs, so I would expect that libvirt
> > > > > > > would
> > > > > > > create each vGPU instance as it's needed. None would be created
> > > > > > > by
> > > > > > > default without user interaction.
> > > > > > >
> > > > > > > Personally I think using a UUID makes sense, but it needs to be
> > > > > > > userspace policy whether that UUID has any implicit meaning like
> > > > > > > matching the VM UUID. Having an index within a UUID bothers me a
> > > > > > > bit,
> > > > > > > but it doesn't seem like too much of a concession to enable the
> > > > > > > use case
> > > > > > > that NVIDIA is trying to achieve. Thanks,
> > > > > > >
> > > > > >
> > > > > > I would prefer to making UUID an optional parameter, while not
> > > > > > tieing
> > > > > > sysfs vgpu naming to UUID. This would be more flexible to different
> > > > > > scenarios where UUID might not be required.
> > > > >
> > > > > Hi Kevin,
> > > > >
> > > > > Happy Chinese New Year!
> > > > >
> > > > > I think having UUID as the vgpu device name will allow us to have an
> > > > > gpu vendor
> > > > > agnostic solution for the upper layer software stack
Re: [Qemu-devel] [RFC PATCH v1 1/1] vGPU core driver : to provide common interface for vGPU.
On Tue, Feb 16, 2016 at 07:40:47AM +, Tian, Kevin wrote:
> > From: Neo Jia [mailto:c...@nvidia.com]
> > Sent: Tuesday, February 16, 2016 3:37 PM
> >
> > On Tue, Feb 16, 2016 at 07:27:09AM +, Tian, Kevin wrote:
> > > > From: Neo Jia [mailto:c...@nvidia.com]
> > > > Sent: Tuesday, February 16, 2016 3:13 PM
> > > >
> > > > On Tue, Feb 16, 2016 at 06:49:30AM +, Tian, Kevin wrote:
> > > > > > From: Alex Williamson [mailto:alex.william...@redhat.com]
> > > > > > Sent: Thursday, February 04, 2016 3:33 AM
> > > > > >
> > > > > > On Wed, 2016-02-03 at 09:28 +0100, Gerd Hoffmann wrote:
> > > > > > > Hi,
> > > > > > >
> > > > > > > > Actually I have a long puzzle in this area. Definitely libvirt
> > > > > > > > will use UUID to
> > > > > > > > mark a VM. And obviously UUID is not recorded within KVM. Then
> > > > > > > > how does
> > > > > > > > libvirt talk to KVM based on UUID? It could be a good reference
> > > > > > > > to this design.
> > > > > > >
> > > > > > > libvirt keeps track which qemu instance belongs to which vm.
> > > > > > > qemu also gets started with "-uuid ...", so one can query qemu via
> > > > > > > monitor ("info uuid") to figure what the uuid is. It is also in
> > > > > > > the
> > > > > > > smbios tables so the guest can see it in the system information
> > > > > > > table.
> > > > > > >
> > > > > > > The uuid is not visible to the kernel though, the kvm kernel
> > > > > > > driver
> > > > > > > doesn't know what the uuid is (and neither does vfio). qemu uses
> > > > > > > file
> > > > > > > handles to talk to both kvm and vfio. qemu notifies both kvm and
> > > > > > > vfio
> > > > > > > about anything relevant events (guest address space changes etc)
> > > > > > > and
> > > > > > > connects file descriptors (eventfd -> irqfd).
> > > > > >
> > > > > > I think the original link to using a VM UUID for the vGPU comes from
> > > > > > NVIDIA having a userspace component which might get launched from a
> > > > > > udev
> > > > > > event as the vGPU is created or the set of vGPUs within that UUID is
> > > > > > started. Using the VM UUID then gives them a way to associate that
> > > > > > userspace process with a VM instance. Maybe it could register with
> > > > > > libvirt for some sort of service provided for the VM, I don't know.
> > > > >
> > > > > Intel doesn't have this requirement. It should be enough as long as
> > > > > libvirt maintains which sysfs vgpu node is associated to a VM UUID.
> > > > >
> > > > > >
> > > > > > > qemu needs a sysfs node as handle to the vfio device, something
> > > > > > > like /sys/devices/virtual/vgpu/. can be a uuid if
> > > > > > > you want
> > > > > > > have it that way, but it could be pretty much anything. The
> > > > > > > sysfs node
> > > > > > > will probably show up as-is in the libvirt xml when assign a vgpu
> > > > > > > to a
> > > > > > > vm. So the name should be something stable (i.e. when using a
> > > > > > > uuid as
> > > > > > > name you should better not generate a new one on each boot).
> > > > > >
> > > > > > Actually I don't think there's really a persistent naming issue,
> > > > > > that's
> > > > > > probably where we diverge from the SR-IOV model. SR-IOV cannot
> > > > > > dynamically add a new VF, it needs to reset the number of VFs to
> > > > > > zero,
> > > > > > then re-allocate all of them up to the new desired count. That has
> > > > > > some
> > > > > > obvious implications. I think with both vendors here, we can
> > > > > > dynamically allocate new vGPUs, so I would expect that libvirt would
> > > > > > create each vGPU instance as it's needed. None would be created by
> > > > > > default without user interaction.
> > > > > >
> > > > > > Personally I think using a UUID makes sense, but it needs to be
> > > > > > userspace policy whether that UUID has any implicit meaning like
> > > > > > matching the VM UUID. Having an index within a UUID bothers me a
> > > > > > bit,
> > > > > > but it doesn't seem like too much of a concession to enable the use
> > > > > > case
> > > > > > that NVIDIA is trying to achieve. Thanks,
> > > > > >
> > > > >
> > > > > I would prefer to making UUID an optional parameter, while not tieing
> > > > > sysfs vgpu naming to UUID. This would be more flexible to different
> > > > > scenarios where UUID might not be required.
> > > >
> > > > Hi Kevin,
> > > >
> > > > Happy Chinese New Year!
> > > >
> > > > I think having UUID as the vgpu device name will allow us to have an
> > > > gpu vendor
> > > > agnostic solution for the upper layer software stack such as QEMU, who
> > > > is
> > > > supposed to open the device.
> > > >
> > >
> > > Qemu can use whatever sysfs path provided to open the device, regardless
> > > of whether there is an UUID within the path...
> > >
> >
> > Hi Kevin,
> >
> > Then it will provide even more benefit of using UUID as libvirt can be
> > implemented as gpu vendor agnostic, right? :-)
> >
> > The UUID can be VM UUID or vGPU group
Re: [Qemu-devel] [RFC PATCH v1 1/1] vGPU core driver : to provide common interface for vGPU.
> From: Neo Jia [mailto:c...@nvidia.com]
> Sent: Tuesday, February 16, 2016 3:37 PM
>
> On Tue, Feb 16, 2016 at 07:27:09AM +, Tian, Kevin wrote:
> > > From: Neo Jia [mailto:c...@nvidia.com]
> > > Sent: Tuesday, February 16, 2016 3:13 PM
> > >
> > > On Tue, Feb 16, 2016 at 06:49:30AM +, Tian, Kevin wrote:
> > > > > From: Alex Williamson [mailto:alex.william...@redhat.com]
> > > > > Sent: Thursday, February 04, 2016 3:33 AM
> > > > >
> > > > > On Wed, 2016-02-03 at 09:28 +0100, Gerd Hoffmann wrote:
> > > > > > Hi,
> > > > > >
> > > > > > > Actually I have a long puzzle in this area. Definitely libvirt
> > > > > > > will use UUID to
> > > > > > > mark a VM. And obviously UUID is not recorded within KVM. Then
> > > > > > > how does
> > > > > > > libvirt talk to KVM based on UUID? It could be a good reference
> > > > > > > to this design.
> > > > > >
> > > > > > libvirt keeps track which qemu instance belongs to which vm.
> > > > > > qemu also gets started with "-uuid ...", so one can query qemu via
> > > > > > monitor ("info uuid") to figure what the uuid is. It is also in the
> > > > > > smbios tables so the guest can see it in the system information
> > > > > > table.
> > > > > >
> > > > > > The uuid is not visible to the kernel though, the kvm kernel driver
> > > > > > doesn't know what the uuid is (and neither does vfio). qemu uses
> > > > > > file
> > > > > > handles to talk to both kvm and vfio. qemu notifies both kvm and
> > > > > > vfio
> > > > > > about anything relevant events (guest address space changes etc) and
> > > > > > connects file descriptors (eventfd -> irqfd).
> > > > >
> > > > > I think the original link to using a VM UUID for the vGPU comes from
> > > > > NVIDIA having a userspace component which might get launched from a
> > > > > udev
> > > > > event as the vGPU is created or the set of vGPUs within that UUID is
> > > > > started. Using the VM UUID then gives them a way to associate that
> > > > > userspace process with a VM instance. Maybe it could register with
> > > > > libvirt for some sort of service provided for the VM, I don't know.
> > > >
> > > > Intel doesn't have this requirement. It should be enough as long as
> > > > libvirt maintains which sysfs vgpu node is associated to a VM UUID.
> > > >
> > > > >
> > > > > > qemu needs a sysfs node as handle to the vfio device, something
> > > > > > like /sys/devices/virtual/vgpu/. can be a uuid if you
> > > > > > want
> > > > > > have it that way, but it could be pretty much anything. The sysfs
> > > > > > node
> > > > > > will probably show up as-is in the libvirt xml when assign a vgpu
> > > > > > to a
> > > > > > vm. So the name should be something stable (i.e. when using a uuid
> > > > > > as
> > > > > > name you should better not generate a new one on each boot).
> > > > >
> > > > > Actually I don't think there's really a persistent naming issue,
> > > > > that's
> > > > > probably where we diverge from the SR-IOV model. SR-IOV cannot
> > > > > dynamically add a new VF, it needs to reset the number of VFs to zero,
> > > > > then re-allocate all of them up to the new desired count. That has
> > > > > some
> > > > > obvious implications. I think with both vendors here, we can
> > > > > dynamically allocate new vGPUs, so I would expect that libvirt would
> > > > > create each vGPU instance as it's needed. None would be created by
> > > > > default without user interaction.
> > > > >
> > > > > Personally I think using a UUID makes sense, but it needs to be
> > > > > userspace policy whether that UUID has any implicit meaning like
> > > > > matching the VM UUID. Having an index within a UUID bothers me a bit,
> > > > > but it doesn't seem like too much of a concession to enable the use
> > > > > case
> > > > > that NVIDIA is trying to achieve. Thanks,
> > > > >
> > > >
> > > > I would prefer to making UUID an optional parameter, while not tieing
> > > > sysfs vgpu naming to UUID. This would be more flexible to different
> > > > scenarios where UUID might not be required.
> > >
> > > Hi Kevin,
> > >
> > > Happy Chinese New Year!
> > >
> > > I think having UUID as the vgpu device name will allow us to have an gpu
> > > vendor
> > > agnostic solution for the upper layer software stack such as QEMU, who is
> > > supposed to open the device.
> > >
> >
> > Qemu can use whatever sysfs path provided to open the device, regardless
> > of whether there is an UUID within the path...
> >
>
> Hi Kevin,
>
> Then it will provide even more benefit of using UUID as libvirt can be
> implemented as gpu vendor agnostic, right? :-)
>
> The UUID can be VM UUID or vGPU group object UUID which really depends on the
> high level software stack, again the benefit is gpu vendor agnostic.
>
There is case where libvirt is not used while another mgmt. stack doesn't use
UUID, e.g. in some Xen scenarios. So it's not about GPU vendor agnostic. It's
about high level mgmt. stack agnostic. That's why we need make UUID as
Re: [Qemu-devel] [RFC PATCH v1 1/1] vGPU core driver : to provide common interface for vGPU.
> From: Neo Jia [mailto:c...@nvidia.com]
> Sent: Tuesday, February 16, 2016 3:13 PM
>
> On Tue, Feb 16, 2016 at 06:49:30AM +, Tian, Kevin wrote:
> > > From: Alex Williamson [mailto:alex.william...@redhat.com]
> > > Sent: Thursday, February 04, 2016 3:33 AM
> > >
> > > On Wed, 2016-02-03 at 09:28 +0100, Gerd Hoffmann wrote:
> > > > Hi,
> > > >
> > > > > Actually I have a long puzzle in this area. Definitely libvirt will
> > > > > use UUID to
> > > > > mark a VM. And obviously UUID is not recorded within KVM. Then how
> > > > > does
> > > > > libvirt talk to KVM based on UUID? It could be a good reference to
> > > > > this design.
> > > >
> > > > libvirt keeps track which qemu instance belongs to which vm.
> > > > qemu also gets started with "-uuid ...", so one can query qemu via
> > > > monitor ("info uuid") to figure what the uuid is. It is also in the
> > > > smbios tables so the guest can see it in the system information table.
> > > >
> > > > The uuid is not visible to the kernel though, the kvm kernel driver
> > > > doesn't know what the uuid is (and neither does vfio). qemu uses file
> > > > handles to talk to both kvm and vfio. qemu notifies both kvm and vfio
> > > > about anything relevant events (guest address space changes etc) and
> > > > connects file descriptors (eventfd -> irqfd).
> > >
> > > I think the original link to using a VM UUID for the vGPU comes from
> > > NVIDIA having a userspace component which might get launched from a udev
> > > event as the vGPU is created or the set of vGPUs within that UUID is
> > > started. Using the VM UUID then gives them a way to associate that
> > > userspace process with a VM instance. Maybe it could register with
> > > libvirt for some sort of service provided for the VM, I don't know.
> >
> > Intel doesn't have this requirement. It should be enough as long as
> > libvirt maintains which sysfs vgpu node is associated to a VM UUID.
> >
> > >
> > > > qemu needs a sysfs node as handle to the vfio device, something
> > > > like /sys/devices/virtual/vgpu/. can be a uuid if you want
> > > > have it that way, but it could be pretty much anything. The sysfs node
> > > > will probably show up as-is in the libvirt xml when assign a vgpu to a
> > > > vm. So the name should be something stable (i.e. when using a uuid as
> > > > name you should better not generate a new one on each boot).
> > >
> > > Actually I don't think there's really a persistent naming issue, that's
> > > probably where we diverge from the SR-IOV model. SR-IOV cannot
> > > dynamically add a new VF, it needs to reset the number of VFs to zero,
> > > then re-allocate all of them up to the new desired count. That has some
> > > obvious implications. I think with both vendors here, we can
> > > dynamically allocate new vGPUs, so I would expect that libvirt would
> > > create each vGPU instance as it's needed. None would be created by
> > > default without user interaction.
> > >
> > > Personally I think using a UUID makes sense, but it needs to be
> > > userspace policy whether that UUID has any implicit meaning like
> > > matching the VM UUID. Having an index within a UUID bothers me a bit,
> > > but it doesn't seem like too much of a concession to enable the use case
> > > that NVIDIA is trying to achieve. Thanks,
> > >
> >
> > I would prefer to making UUID an optional parameter, while not tieing
> > sysfs vgpu naming to UUID. This would be more flexible to different
> > scenarios where UUID might not be required.
>
> Hi Kevin,
>
> Happy Chinese New Year!
>
> I think having UUID as the vgpu device name will allow us to have an gpu
> vendor
> agnostic solution for the upper layer software stack such as QEMU, who is
> supposed to open the device.
>
Qemu can use whatever sysfs path provided to open the device, regardless
of whether there is an UUID within the path...
Thanks
Kevin
Re: [Qemu-devel] [RFC PATCH v1 1/1] vGPU core driver : to provide common interface for vGPU.
On Tue, Feb 16, 2016 at 07:27:09AM +, Tian, Kevin wrote:
> > From: Neo Jia [mailto:c...@nvidia.com]
> > Sent: Tuesday, February 16, 2016 3:13 PM
> >
> > On Tue, Feb 16, 2016 at 06:49:30AM +, Tian, Kevin wrote:
> > > > From: Alex Williamson [mailto:alex.william...@redhat.com]
> > > > Sent: Thursday, February 04, 2016 3:33 AM
> > > >
> > > > On Wed, 2016-02-03 at 09:28 +0100, Gerd Hoffmann wrote:
> > > > > Hi,
> > > > >
> > > > > > Actually I have a long puzzle in this area. Definitely libvirt will
> > > > > > use UUID to
> > > > > > mark a VM. And obviously UUID is not recorded within KVM. Then how
> > > > > > does
> > > > > > libvirt talk to KVM based on UUID? It could be a good reference to
> > > > > > this design.
> > > > >
> > > > > libvirt keeps track which qemu instance belongs to which vm.
> > > > > qemu also gets started with "-uuid ...", so one can query qemu via
> > > > > monitor ("info uuid") to figure what the uuid is. It is also in the
> > > > > smbios tables so the guest can see it in the system information table.
> > > > >
> > > > > The uuid is not visible to the kernel though, the kvm kernel driver
> > > > > doesn't know what the uuid is (and neither does vfio). qemu uses file
> > > > > handles to talk to both kvm and vfio. qemu notifies both kvm and vfio
> > > > > about anything relevant events (guest address space changes etc) and
> > > > > connects file descriptors (eventfd -> irqfd).
> > > >
> > > > I think the original link to using a VM UUID for the vGPU comes from
> > > > NVIDIA having a userspace component which might get launched from a udev
> > > > event as the vGPU is created or the set of vGPUs within that UUID is
> > > > started. Using the VM UUID then gives them a way to associate that
> > > > userspace process with a VM instance. Maybe it could register with
> > > > libvirt for some sort of service provided for the VM, I don't know.
> > >
> > > Intel doesn't have this requirement. It should be enough as long as
> > > libvirt maintains which sysfs vgpu node is associated to a VM UUID.
> > >
> > > >
> > > > > qemu needs a sysfs node as handle to the vfio device, something
> > > > > like /sys/devices/virtual/vgpu/. can be a uuid if you
> > > > > want
> > > > > have it that way, but it could be pretty much anything. The sysfs
> > > > > node
> > > > > will probably show up as-is in the libvirt xml when assign a vgpu to a
> > > > > vm. So the name should be something stable (i.e. when using a uuid as
> > > > > name you should better not generate a new one on each boot).
> > > >
> > > > Actually I don't think there's really a persistent naming issue, that's
> > > > probably where we diverge from the SR-IOV model. SR-IOV cannot
> > > > dynamically add a new VF, it needs to reset the number of VFs to zero,
> > > > then re-allocate all of them up to the new desired count. That has some
> > > > obvious implications. I think with both vendors here, we can
> > > > dynamically allocate new vGPUs, so I would expect that libvirt would
> > > > create each vGPU instance as it's needed. None would be created by
> > > > default without user interaction.
> > > >
> > > > Personally I think using a UUID makes sense, but it needs to be
> > > > userspace policy whether that UUID has any implicit meaning like
> > > > matching the VM UUID. Having an index within a UUID bothers me a bit,
> > > > but it doesn't seem like too much of a concession to enable the use case
> > > > that NVIDIA is trying to achieve. Thanks,
> > > >
> > >
> > > I would prefer to making UUID an optional parameter, while not tieing
> > > sysfs vgpu naming to UUID. This would be more flexible to different
> > > scenarios where UUID might not be required.
> >
> > Hi Kevin,
> >
> > Happy Chinese New Year!
> >
> > I think having UUID as the vgpu device name will allow us to have an gpu
> > vendor
> > agnostic solution for the upper layer software stack such as QEMU, who is
> > supposed to open the device.
> >
>
> Qemu can use whatever sysfs path provided to open the device, regardless
> of whether there is an UUID within the path...
>
Hi Kevin,
Then it will provide even more benefit of using UUID as libvirt can be
implemented as gpu vendor agnostic, right? :-)
The UUID can be VM UUID or vGPU group object UUID which really depends on the
high level software stack, again the benefit is gpu vendor agnostic.
Thanks,
Neo
> Thanks
> Kevin
Re: [Qemu-devel] [RFC PATCH v1 1/1] vGPU core driver : to provide common interface for vGPU.
On Tue, Feb 16, 2016 at 06:49:30AM +, Tian, Kevin wrote:
> > From: Alex Williamson [mailto:alex.william...@redhat.com]
> > Sent: Thursday, February 04, 2016 3:33 AM
> >
> > On Wed, 2016-02-03 at 09:28 +0100, Gerd Hoffmann wrote:
> > > Hi,
> > >
> > > > Actually I have a long puzzle in this area. Definitely libvirt will use
> > > > UUID to
> > > > mark a VM. And obviously UUID is not recorded within KVM. Then how does
> > > > libvirt talk to KVM based on UUID? It could be a good reference to this
> > > > design.
> > >
> > > libvirt keeps track which qemu instance belongs to which vm.
> > > qemu also gets started with "-uuid ...", so one can query qemu via
> > > monitor ("info uuid") to figure what the uuid is. It is also in the
> > > smbios tables so the guest can see it in the system information table.
> > >
> > > The uuid is not visible to the kernel though, the kvm kernel driver
> > > doesn't know what the uuid is (and neither does vfio). qemu uses file
> > > handles to talk to both kvm and vfio. qemu notifies both kvm and vfio
> > > about anything relevant events (guest address space changes etc) and
> > > connects file descriptors (eventfd -> irqfd).
> >
> > I think the original link to using a VM UUID for the vGPU comes from
> > NVIDIA having a userspace component which might get launched from a udev
> > event as the vGPU is created or the set of vGPUs within that UUID is
> > started. Using the VM UUID then gives them a way to associate that
> > userspace process with a VM instance. Maybe it could register with
> > libvirt for some sort of service provided for the VM, I don't know.
>
> Intel doesn't have this requirement. It should be enough as long as
> libvirt maintains which sysfs vgpu node is associated to a VM UUID.
>
> >
> > > qemu needs a sysfs node as handle to the vfio device, something
> > > like /sys/devices/virtual/vgpu/. can be a uuid if you want
> > > have it that way, but it could be pretty much anything. The sysfs node
> > > will probably show up as-is in the libvirt xml when assign a vgpu to a
> > > vm. So the name should be something stable (i.e. when using a uuid as
> > > name you should better not generate a new one on each boot).
> >
> > Actually I don't think there's really a persistent naming issue, that's
> > probably where we diverge from the SR-IOV model. SR-IOV cannot
> > dynamically add a new VF, it needs to reset the number of VFs to zero,
> > then re-allocate all of them up to the new desired count. That has some
> > obvious implications. I think with both vendors here, we can
> > dynamically allocate new vGPUs, so I would expect that libvirt would
> > create each vGPU instance as it's needed. None would be created by
> > default without user interaction.
> >
> > Personally I think using a UUID makes sense, but it needs to be
> > userspace policy whether that UUID has any implicit meaning like
> > matching the VM UUID. Having an index within a UUID bothers me a bit,
> > but it doesn't seem like too much of a concession to enable the use case
> > that NVIDIA is trying to achieve. Thanks,
> >
>
> I would prefer to making UUID an optional parameter, while not tieing
> sysfs vgpu naming to UUID. This would be more flexible to different
> scenarios where UUID might not be required.
Hi Kevin,
Happy Chinese New Year!
I think having UUID as the vgpu device name will allow us to have an gpu vendor
agnostic solution for the upper layer software stack such as QEMU, who is
supposed to open the device.
Thanks,
Neo
>
> Thanks
> Kevin
Re: [Qemu-devel] [RFC PATCH v1 1/1] vGPU core driver : to provide common interface for vGPU.
> From: Alex Williamson [mailto:alex.william...@redhat.com]
> Sent: Thursday, February 04, 2016 3:33 AM
>
> On Wed, 2016-02-03 at 09:28 +0100, Gerd Hoffmann wrote:
> > Hi,
> >
> > > Actually I have a long puzzle in this area. Definitely libvirt will use
> > > UUID to
> > > mark a VM. And obviously UUID is not recorded within KVM. Then how does
> > > libvirt talk to KVM based on UUID? It could be a good reference to this
> > > design.
> >
> > libvirt keeps track which qemu instance belongs to which vm.
> > qemu also gets started with "-uuid ...", so one can query qemu via
> > monitor ("info uuid") to figure what the uuid is. It is also in the
> > smbios tables so the guest can see it in the system information table.
> >
> > The uuid is not visible to the kernel though, the kvm kernel driver
> > doesn't know what the uuid is (and neither does vfio). qemu uses file
> > handles to talk to both kvm and vfio. qemu notifies both kvm and vfio
> > about anything relevant events (guest address space changes etc) and
> > connects file descriptors (eventfd -> irqfd).
>
> I think the original link to using a VM UUID for the vGPU comes from
> NVIDIA having a userspace component which might get launched from a udev
> event as the vGPU is created or the set of vGPUs within that UUID is
> started. Using the VM UUID then gives them a way to associate that
> userspace process with a VM instance. Maybe it could register with
> libvirt for some sort of service provided for the VM, I don't know.
Intel doesn't have this requirement. It should be enough as long as
libvirt maintains which sysfs vgpu node is associated to a VM UUID.
>
> > qemu needs a sysfs node as handle to the vfio device, something
> > like /sys/devices/virtual/vgpu/. can be a uuid if you want
> > have it that way, but it could be pretty much anything. The sysfs node
> > will probably show up as-is in the libvirt xml when assign a vgpu to a
> > vm. So the name should be something stable (i.e. when using a uuid as
> > name you should better not generate a new one on each boot).
>
> Actually I don't think there's really a persistent naming issue, that's
> probably where we diverge from the SR-IOV model. SR-IOV cannot
> dynamically add a new VF, it needs to reset the number of VFs to zero,
> then re-allocate all of them up to the new desired count. That has some
> obvious implications. I think with both vendors here, we can
> dynamically allocate new vGPUs, so I would expect that libvirt would
> create each vGPU instance as it's needed. None would be created by
> default without user interaction.
>
> Personally I think using a UUID makes sense, but it needs to be
> userspace policy whether that UUID has any implicit meaning like
> matching the VM UUID. Having an index within a UUID bothers me a bit,
> but it doesn't seem like too much of a concession to enable the use case
> that NVIDIA is trying to achieve. Thanks,
>
I would prefer to making UUID an optional parameter, while not tieing
sysfs vgpu naming to UUID. This would be more flexible to different
scenarios where UUID might not be required.
Thanks
Kevin
Re: [Qemu-devel] [RFC PATCH v1 1/1] vGPU core driver : to provide common interface for vGPU.
Hi,
> Actually I have a long puzzle in this area. Definitely libvirt will use UUID
> to
> mark a VM. And obviously UUID is not recorded within KVM. Then how does
> libvirt talk to KVM based on UUID? It could be a good reference to this
> design.
libvirt keeps track which qemu instance belongs to which vm.
qemu also gets started with "-uuid ...", so one can query qemu via
monitor ("info uuid") to figure what the uuid is. It is also in the
smbios tables so the guest can see it in the system information table.
The uuid is not visible to the kernel though, the kvm kernel driver
doesn't know what the uuid is (and neither does vfio). qemu uses file
handles to talk to both kvm and vfio. qemu notifies both kvm and vfio
about anything relevant events (guest address space changes etc) and
connects file descriptors (eventfd -> irqfd).
qemu needs a sysfs node as handle to the vfio device, something
like /sys/devices/virtual/vgpu/. can be a uuid if you want
have it that way, but it could be pretty much anything. The sysfs node
will probably show up as-is in the libvirt xml when assign a vgpu to a
vm. So the name should be something stable (i.e. when using a uuid as
name you should better not generate a new one on each boot).
cheers,
Gerd
Re: [Qemu-devel] [RFC PATCH v1 1/1] vGPU core driver : to provide common interface for vGPU.
On 2/3/2016 11:26 AM, Tian, Kevin wrote: [...] * @vgpu_create:Called to allocate basic resouces in graphics * driver for a particular vgpu. * @dev: physical pci device structure on which vgpu *should be created * @uuid: uuid for which VM it is intended to * @instance: vgpu instance in that VM * @vgpu_id: This represents the type of vgpu to be *created * Returns integer: success (0) or error (< 0) Specifically for Intel GVT-g we didn't hard partition resource among vGPUs. Instead we allow user to accurately control how many physical resources are allocated to a vGPU. So this interface should be extensible to allow vendor specific resource control. This interface forwards the create request to vendor/GPU driver informing about which physical GPU this request is intended for and the type of vGPU. Then its vendor/GPU driver's responsibility to do resources allocation and manage resources in their own driver. However the current parameter definition disallows resource configuration information passed from user. As I said, Intel GVT-g doesn't do static allocation based on type. We provide flexibility to user for fine-grained resource management. int (*vgpu_create)(struct pci_dev *dev, uuid_le uuid, - uint32_t instance, uint32_t vgpu_id); + uint32_t instance, char *vgpu_params); If we change integer vgpu_id parameter to char *vgpu_param then GPU driver can have multiple parameters. Suppose there is a GPU at :85:00.0, then to create vgpu: # echo "::" > /sys/bus/pci/devices/\:85\:00.0/vgpu_create Common vgpu module will not parse vgpu_params string, it will be forwarded to GPU driver, then its GPU driver's responsibility to parse the string and act accordingly. This should give flexibility to have multiple parameters for GPU driver. * * Physical GPU that support vGPU should be register with vgpu module with * gpu_device_ops structure. */ Also it'd be good design to allow extensible usages, such as statistics, and other vendor specific control knobs (e.g. foreground/background VM switch in Intel GVT-g, etc.) Can you elaborate on what other control knobs that would be needed? Some examples: - foreground/background VM switch - resource query - various statistics info - virtual monitor configuration - ... Since this vgpu core driver will become the central point for all vgpu management, we need provide an easy way for vendor specific extension, e.g. exposing above callbacks as sysfs nodes and then vendor can create its own extensions under subdirectory (/intel, /nvidia, ...). Ok. Makes sense. Are these parameters per physical device or per vgpu device? Adding attribute groups to gpu_device_ops structure would provide a way to add vendor specific extensions. I think we need two types of attributes here, per physical device and per vgpu device. Right? const struct attribute_group **dev_groups; const struct attribute_group **vgpu_groups; Thanks, Kirti. Thanks Kevin
Re: [Qemu-devel] [RFC PATCH v1 1/1] vGPU core driver : to provide common interface for vGPU.
> From: Kirti Wankhede > Sent: Wednesday, February 03, 2016 9:21 PM > > > On 2/3/2016 11:26 AM, Tian, Kevin wrote: > [...] > > * @vgpu_create:Called to allocate basic resouces in graphics > > * driver for a particular vgpu. > > * @dev: physical pci device structure on which > > vgpu > > *should be created > > * @uuid: uuid for which VM it is intended to > > * @instance: vgpu instance in that VM > > * @vgpu_id: This represents the type of vgpu > > to be > > *created > > * Returns integer: success (0) or error (< 0) > >>> > >>> Specifically for Intel GVT-g we didn't hard partition resource among > >>> vGPUs. > >>> Instead we allow user to accurately control how many physical resources > >>> are allocated to a vGPU. So this interface should be extensible to allow > >>> vendor specific resource control. > >>> > >> > >> This interface forwards the create request to vendor/GPU driver > >> informing about which physical GPU this request is intended for and the > >> type of vGPU. Then its vendor/GPU driver's responsibility to do > >> resources allocation and manage resources in their own driver. > > > > However the current parameter definition disallows resource configuration > > information passed from user. As I said, Intel GVT-g doesn't do static > > allocation based on type. We provide flexibility to user for fine-grained > > resource management. > > > > int (*vgpu_create)(struct pci_dev *dev, uuid_le uuid, > - uint32_t instance, uint32_t vgpu_id); > + uint32_t instance, char *vgpu_params); > > If we change integer vgpu_id parameter to char *vgpu_param then GPU > driver can have multiple parameters. > > Suppose there is a GPU at :85:00.0, then to create vgpu: > # echo "::" > > /sys/bus/pci/devices/\:85\:00.0/vgpu_create > > Common vgpu module will not parse vgpu_params string, it will be > forwarded to GPU driver, then its GPU driver's responsibility to parse > the string and act accordingly. This should give flexibility to have > multiple parameters for GPU driver. Yes, this way it would work. > > > * > > * Physical GPU that support vGPU should be register with vgpu > > module with > > * gpu_device_ops structure. > > */ > > > >>> > >>> Also it'd be good design to allow extensible usages, such as statistics, > >>> and > >>> other vendor specific control knobs (e.g. foreground/background VM switch > >>> in Intel GVT-g, etc.) > >>> > >> > >> Can you elaborate on what other control knobs that would be needed? > >> > > > > Some examples: > > > > - foreground/background VM switch > > - resource query > > - various statistics info > > - virtual monitor configuration > > - ... > > > > Since this vgpu core driver will become the central point for all vgpu > > management, we need provide an easy way for vendor specific extension, > > e.g. exposing above callbacks as sysfs nodes and then vendor can create > > its own extensions under subdirectory (/intel, /nvidia, ...). > > > > > > Ok. Makes sense. > > Are these parameters per physical device or per vgpu device? some for physical, others for vgpu. > > Adding attribute groups to gpu_device_ops structure would provide a way > to add vendor specific extensions. I think we need two types of > attributes here, per physical device and per vgpu device. Right? > > const struct attribute_group **dev_groups; > const struct attribute_group **vgpu_groups; > > > Agree. Thanks Kevin
Re: [Qemu-devel] [RFC PATCH v1 1/1] vGPU core driver : to provide common interface for vGPU.
On Wed, 2016-02-03 at 09:28 +0100, Gerd Hoffmann wrote:
> Hi,
>
> > Actually I have a long puzzle in this area. Definitely libvirt will use
> > UUID to
> > mark a VM. And obviously UUID is not recorded within KVM. Then how does
> > libvirt talk to KVM based on UUID? It could be a good reference to this
> > design.
>
> libvirt keeps track which qemu instance belongs to which vm.
> qemu also gets started with "-uuid ...", so one can query qemu via
> monitor ("info uuid") to figure what the uuid is. It is also in the
> smbios tables so the guest can see it in the system information table.
>
> The uuid is not visible to the kernel though, the kvm kernel driver
> doesn't know what the uuid is (and neither does vfio). qemu uses file
> handles to talk to both kvm and vfio. qemu notifies both kvm and vfio
> about anything relevant events (guest address space changes etc) and
> connects file descriptors (eventfd -> irqfd).
I think the original link to using a VM UUID for the vGPU comes from
NVIDIA having a userspace component which might get launched from a udev
event as the vGPU is created or the set of vGPUs within that UUID is
started. Using the VM UUID then gives them a way to associate that
userspace process with a VM instance. Maybe it could register with
libvirt for some sort of service provided for the VM, I don't know.
> qemu needs a sysfs node as handle to the vfio device, something
> like /sys/devices/virtual/vgpu/. can be a uuid if you want
> have it that way, but it could be pretty much anything. The sysfs node
> will probably show up as-is in the libvirt xml when assign a vgpu to a
> vm. So the name should be something stable (i.e. when using a uuid as
> name you should better not generate a new one on each boot).
Actually I don't think there's really a persistent naming issue, that's
probably where we diverge from the SR-IOV model. SR-IOV cannot
dynamically add a new VF, it needs to reset the number of VFs to zero,
then re-allocate all of them up to the new desired count. That has some
obvious implications. I think with both vendors here, we can
dynamically allocate new vGPUs, so I would expect that libvirt would
create each vGPU instance as it's needed. None would be created by
default without user interaction.
Personally I think using a UUID makes sense, but it needs to be
userspace policy whether that UUID has any implicit meaning like
matching the VM UUID. Having an index within a UUID bothers me a bit,
but it doesn't seem like too much of a concession to enable the use case
that NVIDIA is trying to achieve. Thanks,
Alex
Re: [Qemu-devel] [RFC PATCH v1 1/1] vGPU core driver : to provide common interface for vGPU.
> From: Alex Williamson [mailto:alex.william...@redhat.com] > Sent: Wednesday, February 03, 2016 1:11 AM > > On Tue, 2016-02-02 at 00:31 -0800, Neo Jia wrote: > > On Tue, Feb 02, 2016 at 08:18:44AM +, Tian, Kevin wrote: > > > > From: Neo Jia [mailto:c...@nvidia.com] > > > > Sent: Tuesday, February 02, 2016 4:13 PM > > > > > > > > On Tue, Feb 02, 2016 at 09:00:43AM +0100, Gerd Hoffmann wrote: > > > > > Hi, > > > > > > > > > > > And for UUID, I remember Alex had a concern on using it in kernel. > > > > > > Honestly speaking I don't have a good idea here. In Xen side there > > > > > > is a VM ID > > > > > > which can be easily used as the index. But for KVM, what would be > > > > > > the best > > > > > > identifier to associate with a VM? > > > > > > > > > > The vgpu code doesn't need to associate the vgpu device with a vm in > > > > > the > > > > > first place. You get all guest address space information from qemu, > > > > > via > > > > > vfio iommu interface. > > > > > > > > > > When qemu does't use kvm (tcg mode), things should still work fine. > > > > > Using vfio-based vgpu devices with non-qemu apps (some kind of test > > > > > suite for example) should work fine too. > > > > > > > > Hi Gerd and Kevin, > > > > > > > > I thought Alex had agreed with the UUID as long as it is not tied with > > > > VM, > > > > probably it is just his comment gets lost in our previous long email > > > > thread. > > > > > > > > > > I think the point is... what is the value to introduce a UUID here? If > > > what Gerd describes is enough, we can simply invent a vgpu ID which > > > is returned at vgpu_create, and then used as the index for other > > > interfaces. > > > > > > > Hi Kevin, > > > > It can just be a plain UUID, and the meaning of the UUID is up to upper > > layer SW, for > > example with libvirt, you can create a new "vgpu group" object representing > > a > > list of vgpu device. so the UUID will be the input on vgpu_create instead of > > return value. > > Jumping in at th end, but yes, this was my thinking. A UUID is a > perfectly fine name for a vgpu, but it should be user policy whether > than UUID matches a VM definition or is simply an arbitrary grouping of > vgpus. > well I'm still trying to understand the necessity here. Is there any requirement that vgpu driver needs to use UUID itself? If not, we can still invent any simple format in vgpu driver itself, as Gerd mentioned, w/o dependency on an user input. Then libvirt can maintain UUID->vgpu ID internally. Actually I have a long puzzle in this area. Definitely libvirt will use UUID to mark a VM. And obviously UUID is not recorded within KVM. Then how does libvirt talk to KVM based on UUID? It could be a good reference to this design. Thanks Kevin
Re: [Qemu-devel] [RFC PATCH v1 1/1] vGPU core driver : to provide common interface for vGPU.
> From: Kirti Wankhede > Sent: Tuesday, February 02, 2016 5:25 PM > > On 2/2/2016 1:12 PM, Tian, Kevin wrote: > >> From: Kirti Wankhede [mailto:kwankh...@nvidia.com] > >> Sent: Tuesday, February 02, 2016 9:48 AM > >> > >> Resending this mail again, somehow my previous mail didn't reached every > >> to everyone's inbox. > >> > >> On 2/2/2016 3:16 AM, Kirti Wankhede wrote: > >>> Design for vGPU Driver: > >>> Main purpose of vGPU driver is to provide a common interface for vGPU > >>> management that can be used by differnt GPU drivers. > > > > Thanks for composing this design which is a good start. > > > >>> > >>> This module would provide a generic interface to create the device, add > >>> it to vGPU bus, add device to IOMMU group and then add it to vfio group. > >>> > >>> High Level block diagram: > >>> > >>> > >>> +--+vgpu_register_driver()+---+ > >>> | __init() +->+ | > >>> | | | | > >>> | +<-+vgpu.ko| > >>> | vfio_vgpu.ko | probe()/remove() | | > >>> | |+-+ +-+ > >>> +--+| +---+---+ | > >>> | ^ | > >>> | callback| | > >>> | +---++| > >>> | |vgpu_register_device() | > >>> | ||| > >>> +---^-+-++-+--+-+ > >>> | nvidia.ko || i915.ko | > >>> | ||| > >>> +---+++ > >>> > >>> vGPU driver provides two types of registration interfaces: > > > > Looks you missed callbacks which vgpu.ko provides to vfio_vgpu.ko, > > e.g. to retrieve basic region/resource info, etc... > > > > Basic region info or resource info would come from GPU driver. So > retrieving such info should be the part of GPU driver interface. Like I > mentioned we need to enhance these interfaces during development as and > when we find it useful. > vfio_vgpu.ko gets dev pointer from which it can reach to vgpu_device > structure and then it can use GPU driver interface directly to retrieve > such information from GPU driver. Thanks for confirmation. Want to make sure we are on the same page since those callbacks are not listed in current proposal yet. > > This RFC is to focus more on different modules and its structures, how > those modules would be inter-linked with each other and have a flexible > design to keep the scope for enhancements. > > We have identified three modules: > > * vgpu.ko - vGPU core driver that provide registration interface for GPU > driver and vGPU VFIO driver, responsible for creating vGPU devices and > providing management interface for vGPU devices. > * vfio_vgpu.ko - vGPU VFIO driver for vGPU device, provides VFIO > interface that is used by QEMU. > * vfio_iommu_type1_vgpu.ko - IOMMU TYPE1 driver supporting the IOMMU > TYPE1 v1 and v2 interface. > > The above block diagram gives an overview how vgpu.ko, vfio_vgpu.ko and > GPU drivers would be inter-linked with each other. This part is clear. > >>>* @vgpu_create:Called to allocate basic resouces in graphics > >>>* driver for a particular vgpu. > >>>* @dev: physical pci device structure on which > >>> vgpu > >>>*should be created > >>>* @uuid: uuid for which VM it is intended to > >>>* @instance: vgpu instance in that VM > >>>* @vgpu_id: This represents the type of vgpu to be > >>>*created > >>>* Returns integer: success (0) or error (< 0) > > > > Specifically for Intel GVT-g we didn't hard partition resource among vGPUs. > > Instead we allow user to accurately control how many physical resources > > are allocated to a vGPU. So this interface should be extensible to allow > > vendor specific resource control. > > > > This interface forwards the create request to vendor/GPU driver > informing about which physical GPU this request is intended for and the > type of vGPU. Then its vendor/GPU driver's responsibility to do > resources allocation and manage resources in their own driver. However the current parameter definition disallows resource configuration information passed from user. As I said, Intel GVT-g doesn't do static allocation based on type. We provide flexibility to user for fine-grained resource management. > >>>* > >>>
Re: [Qemu-devel] [RFC PATCH v1 1/1] vGPU core driver : to provide common interface for vGPU.
On Tue, Feb 02, 2016 at 09:00:43AM +0100, Gerd Hoffmann wrote: > Hi, > > > And for UUID, I remember Alex had a concern on using it in kernel. > > Honestly speaking I don't have a good idea here. In Xen side there is a VM > > ID > > which can be easily used as the index. But for KVM, what would be the best > > identifier to associate with a VM? > > The vgpu code doesn't need to associate the vgpu device with a vm in the > first place. You get all guest address space information from qemu, via > vfio iommu interface. > > When qemu does't use kvm (tcg mode), things should still work fine. > Using vfio-based vgpu devices with non-qemu apps (some kind of test > suite for example) should work fine too. Hi Gerd and Kevin, I thought Alex had agreed with the UUID as long as it is not tied with VM, probably it is just his comment gets lost in our previous long email thread. Thanks, Neo > > cheers, > Gerd >
Re: [Qemu-devel] [RFC PATCH v1 1/1] vGPU core driver : to provide common interface for vGPU.
> From: Neo Jia [mailto:c...@nvidia.com] > Sent: Tuesday, February 02, 2016 4:13 PM > > On Tue, Feb 02, 2016 at 09:00:43AM +0100, Gerd Hoffmann wrote: > > Hi, > > > > > And for UUID, I remember Alex had a concern on using it in kernel. > > > Honestly speaking I don't have a good idea here. In Xen side there is a > > > VM ID > > > which can be easily used as the index. But for KVM, what would be the best > > > identifier to associate with a VM? > > > > The vgpu code doesn't need to associate the vgpu device with a vm in the > > first place. You get all guest address space information from qemu, via > > vfio iommu interface. > > > > When qemu does't use kvm (tcg mode), things should still work fine. > > Using vfio-based vgpu devices with non-qemu apps (some kind of test > > suite for example) should work fine too. > > Hi Gerd and Kevin, > > I thought Alex had agreed with the UUID as long as it is not tied with VM, > probably it is just his comment gets lost in our previous long email thread. > I think the point is... what is the value to introduce a UUID here? If what Gerd describes is enough, we can simply invent a vgpu ID which is returned at vgpu_create, and then used as the index for other interfaces. But I still need to think about whether there's value to have a VM identifier within vgpu core driver, especially regarding to how this vgpu core driver connects to KVM hypervisor or other hypervisor. I saw another long thread about that part. Jike has started his vacation now. I'll follow up with it tomorrow. Thanks Kevin
Re: [Qemu-devel] [RFC PATCH v1 1/1] vGPU core driver : to provide common interface for vGPU.
On Di, 2016-02-02 at 00:13 -0800, Neo Jia wrote:
> On Tue, Feb 02, 2016 at 09:00:43AM +0100, Gerd Hoffmann wrote:
> > Hi,
> >
> > > And for UUID, I remember Alex had a concern on using it in kernel.
> > > Honestly speaking I don't have a good idea here. In Xen side there is a
> > > VM ID
> > > which can be easily used as the index. But for KVM, what would be the best
> > > identifier to associate with a VM?
> >
> > The vgpu code doesn't need to associate the vgpu device with a vm in the
> > first place. You get all guest address space information from qemu, via
> > vfio iommu interface.
> >
> > When qemu does't use kvm (tcg mode), things should still work fine.
> > Using vfio-based vgpu devices with non-qemu apps (some kind of test
> > suite for example) should work fine too.
>
> Hi Gerd and Kevin,
>
> I thought Alex had agreed with the UUID as long as it is not tied with VM,
> probably it is just his comment gets lost in our previous long email thread.
As long as it isn't tied to a VM it is fine indeed. We'll need names
for the devices, and using a uuid certainly is one option. Could also
be a simple "vgpu${index}". A more descriptive name such as
"nvidia-grid-${index}" or "igd-skl-${index}". Or just a free-form
string where the driver can fill whatever it wants (including a uuid).
cheers,
Gerd
Re: [Qemu-devel] [RFC PATCH v1 1/1] vGPU core driver : to provide common interface for vGPU.
Hi, > And for UUID, I remember Alex had a concern on using it in kernel. > Honestly speaking I don't have a good idea here. In Xen side there is a VM ID > which can be easily used as the index. But for KVM, what would be the best > identifier to associate with a VM? The vgpu code doesn't need to associate the vgpu device with a vm in the first place. You get all guest address space information from qemu, via vfio iommu interface. When qemu does't use kvm (tcg mode), things should still work fine. Using vfio-based vgpu devices with non-qemu apps (some kind of test suite for example) should work fine too. cheers, Gerd
Re: [Qemu-devel] [RFC PATCH v1 1/1] vGPU core driver : to provide common interface for vGPU.
On Tue, Feb 02, 2016 at 08:18:44AM +, Tian, Kevin wrote: > > From: Neo Jia [mailto:c...@nvidia.com] > > Sent: Tuesday, February 02, 2016 4:13 PM > > > > On Tue, Feb 02, 2016 at 09:00:43AM +0100, Gerd Hoffmann wrote: > > > Hi, > > > > > > > And for UUID, I remember Alex had a concern on using it in kernel. > > > > Honestly speaking I don't have a good idea here. In Xen side there is a > > > > VM ID > > > > which can be easily used as the index. But for KVM, what would be the > > > > best > > > > identifier to associate with a VM? > > > > > > The vgpu code doesn't need to associate the vgpu device with a vm in the > > > first place. You get all guest address space information from qemu, via > > > vfio iommu interface. > > > > > > When qemu does't use kvm (tcg mode), things should still work fine. > > > Using vfio-based vgpu devices with non-qemu apps (some kind of test > > > suite for example) should work fine too. > > > > Hi Gerd and Kevin, > > > > I thought Alex had agreed with the UUID as long as it is not tied with VM, > > probably it is just his comment gets lost in our previous long email thread. > > > > I think the point is... what is the value to introduce a UUID here? If > what Gerd describes is enough, we can simply invent a vgpu ID which > is returned at vgpu_create, and then used as the index for other > interfaces. > Hi Kevin, It can just be a plain UUID, and the meaning of the UUID is up to upper layer SW, for example with libvirt, you can create a new "vgpu group" object representing a list of vgpu device. so the UUID will be the input on vgpu_create instead of return value. For the TCG mode, this should just work as long as libvirt can create the proper internal objects there plus other vfio iommu interface Gerd has called out, although the vector->kvm_interrupt part might need some tweaks. Thanks, Neo > But I still need to think about whether there's value to have a VM > identifier within vgpu core driver, especially regarding to how this > vgpu core driver connects to KVM hypervisor or other hypervisor. > I saw another long thread about that part. Jike has started his > vacation now. I'll follow up with it tomorrow. > > Thanks > Kevin
Re: [Qemu-devel] [RFC PATCH v1 1/1] vGPU core driver : to provide common interface for vGPU.
On 2/2/2016 1:12 PM, Tian, Kevin wrote: From: Kirti Wankhede [mailto:kwankh...@nvidia.com] Sent: Tuesday, February 02, 2016 9:48 AM Resending this mail again, somehow my previous mail didn't reached every to everyone's inbox. On 2/2/2016 3:16 AM, Kirti Wankhede wrote: Design for vGPU Driver: Main purpose of vGPU driver is to provide a common interface for vGPU management that can be used by differnt GPU drivers. Thanks for composing this design which is a good start. This module would provide a generic interface to create the device, add it to vGPU bus, add device to IOMMU group and then add it to vfio group. High Level block diagram: +--+vgpu_register_driver()+---+ | __init() +->+ | | | | | | +<-+vgpu.ko| | vfio_vgpu.ko | probe()/remove() | | | |+-+ +-+ +--+| +---+---+ | | ^ | | callback| | | +---++| | |vgpu_register_device() | | ||| +---^-+-++-+--+-+ | nvidia.ko || i915.ko | | ||| +---+++ vGPU driver provides two types of registration interfaces: Looks you missed callbacks which vgpu.ko provides to vfio_vgpu.ko, e.g. to retrieve basic region/resource info, etc... Basic region info or resource info would come from GPU driver. So retrieving such info should be the part of GPU driver interface. Like I mentioned we need to enhance these interfaces during development as and when we find it useful. vfio_vgpu.ko gets dev pointer from which it can reach to vgpu_device structure and then it can use GPU driver interface directly to retrieve such information from GPU driver. This RFC is to focus more on different modules and its structures, how those modules would be inter-linked with each other and have a flexible design to keep the scope for enhancements. We have identified three modules: * vgpu.ko - vGPU core driver that provide registration interface for GPU driver and vGPU VFIO driver, responsible for creating vGPU devices and providing management interface for vGPU devices. * vfio_vgpu.ko - vGPU VFIO driver for vGPU device, provides VFIO interface that is used by QEMU. * vfio_iommu_type1_vgpu.ko - IOMMU TYPE1 driver supporting the IOMMU TYPE1 v1 and v2 interface. The above block diagram gives an overview how vgpu.ko, vfio_vgpu.ko and GPU drivers would be inter-linked with each other. Also for GPU driver interfaces, better to identify the caller. E.g. it's easy to understand life-cycle management would come from sysfs by mgmt. stack like libvirt. What about @read and @write? what's the connection between this vgpu core driver and specific hypervisor? etc. Better to connect all necessary dots so we can refine all necessary requirements on this proposal. read and write calls are for PCI CFG space amd MMIO space read/write. Read/write access request from QEMU is passed to GPU driver through GPU driver interface. [...] 2. GPU driver interface /** * struct gpu_device_ops - Structure to be registered for each physical GPU to * register the device to vgpu module. * * @owner: The module owner. * @vgpu_supported_config: Called to get information about supported * vgpu types. * @dev : pci device structure of physical GPU. * @config: should return string listing supported * config * Returns integer: success (0) or error (< 0) * @vgpu_create:Called to allocate basic resouces in graphics * driver for a particular vgpu. * @dev: physical pci device structure on which vgpu *should be created * @uuid: uuid for which VM it is intended to * @instance: vgpu instance in that VM * @vgpu_id: This represents the type of vgpu to be *created * Returns integer: success (0) or error (< 0) Specifically for Intel GVT-g we didn't hard partition resource among vGPUs. Instead we allow user to accurately control how many physical resources are allocated to a vGPU. So this interface should be extensible to allow vendor
Re: [Qemu-devel] [RFC PATCH v1 1/1] vGPU core driver : to provide common interface for vGPU.
On Tue, 2016-02-02 at 00:31 -0800, Neo Jia wrote: > On Tue, Feb 02, 2016 at 08:18:44AM +, Tian, Kevin wrote: > > > From: Neo Jia [mailto:c...@nvidia.com] > > > Sent: Tuesday, February 02, 2016 4:13 PM > > > > > > On Tue, Feb 02, 2016 at 09:00:43AM +0100, Gerd Hoffmann wrote: > > > > Hi, > > > > > > > > > And for UUID, I remember Alex had a concern on using it in kernel. > > > > > Honestly speaking I don't have a good idea here. In Xen side there is > > > > > a VM ID > > > > > which can be easily used as the index. But for KVM, what would be the > > > > > best > > > > > identifier to associate with a VM? > > > > > > > > The vgpu code doesn't need to associate the vgpu device with a vm in the > > > > first place. You get all guest address space information from qemu, via > > > > vfio iommu interface. > > > > > > > > When qemu does't use kvm (tcg mode), things should still work fine. > > > > Using vfio-based vgpu devices with non-qemu apps (some kind of test > > > > suite for example) should work fine too. > > > > > > Hi Gerd and Kevin, > > > > > > I thought Alex had agreed with the UUID as long as it is not tied with VM, > > > probably it is just his comment gets lost in our previous long email > > > thread. > > > > > > > I think the point is... what is the value to introduce a UUID here? If > > what Gerd describes is enough, we can simply invent a vgpu ID which > > is returned at vgpu_create, and then used as the index for other > > interfaces. > > > > Hi Kevin, > > It can just be a plain UUID, and the meaning of the UUID is up to upper layer > SW, for > example with libvirt, you can create a new "vgpu group" object representing a > list of vgpu device. so the UUID will be the input on vgpu_create instead of > return value. Jumping in at th end, but yes, this was my thinking. A UUID is a perfectly fine name for a vgpu, but it should be user policy whether than UUID matches a VM definition or is simply an arbitrary grouping of vgpus. > For the TCG mode, this should just work as long as libvirt can create the > proper > internal objects there plus other vfio iommu interface Gerd has called out, > although the vector->kvm_interrupt part might need some tweaks. Interrupts should be eventfds and whether the eventfd triggers into userspace or into an irqfd in KVM should be completely transparent to the vgpu code, just as is done with vfio today. Thanks, Alex
Re: [Qemu-devel] [RFC PATCH v1 1/1] vGPU core driver : to provide common interface for vGPU.
Resending this mail again, somehow my previous mail didn't reached every
to everyone's inbox.
On 2/2/2016 3:16 AM, Kirti Wankhede wrote:
Design for vGPU Driver:
Main purpose of vGPU driver is to provide a common interface for vGPU
management that can be used by differnt GPU drivers.
This module would provide a generic interface to create the device, add
it to vGPU bus, add device to IOMMU group and then add it to vfio group.
High Level block diagram:
+--+vgpu_register_driver()+---+
| __init() +->+ |
| | | |
| +<-+vgpu.ko|
| vfio_vgpu.ko | probe()/remove() | |
| |+-+ +-+
+--+| +---+---+ |
| ^ |
| callback| |
| +---++|
| |vgpu_register_device() |
| |||
+---^-+-++-+--+-+
| nvidia.ko || i915.ko |
| |||
+---+++
vGPU driver provides two types of registration interfaces:
1. Registration interface for vGPU bus driver:
/**
* struct vgpu_driver - vGPU device driver
* @name: driver name
* @probe: called when new device created
* @remove: called when device removed
* @driver: device driver structure
*
**/
struct vgpu_driver {
const char *name;
int (*probe) (struct device *dev);
void (*remove) (struct device *dev);
struct device_driverdriver;
};
int vgpu_register_driver(struct vgpu_driver *drv, struct module *owner);
void vgpu_unregister_driver(struct vgpu_driver *drv);
VFIO bus driver for vgpu, should use this interface to register with
vGPU driver. With this, VFIO bus driver for vGPU devices is responsible
to add vGPU device to VFIO group.
2. GPU driver interface
/**
* struct gpu_device_ops - Structure to be registered for each physical
GPU to
* register the device to vgpu module.
*
* @owner: The module owner.
* @vgpu_supported_config: Called to get information about supported
* vgpu types.
* @dev : pci device structure of physical GPU.
* @config: should return string listing supported
* config
* Returns integer: success (0) or error (< 0)
* @vgpu_create:Called to allocate basic resouces in graphics
* driver for a particular vgpu.
* @dev: physical pci device structure on which vgpu
*should be created
* @uuid: uuid for which VM it is intended to
* @instance: vgpu instance in that VM
* @vgpu_id: This represents the type of vgpu to be
*created
* Returns integer: success (0) or error (< 0)
* @vgpu_destroy: Called to free resources in graphics driver for
* a vgpu instance of that VM.
* @dev: physical pci device structure to which
* this vgpu points to.
* @uuid: uuid for which the vgpu belongs to.
* @instance: vgpu instance in that VM
* Returns integer: success (0) or error (< 0)
* If VM is running and vgpu_destroy is called that
* means the vGPU is being hotunpluged. Return error
* if VM is running and graphics driver doesn't
* support vgpu hotplug.
* @vgpu_start: Called to do initiate vGPU initialization
* process in graphics driver when VM boots before
* qemu starts.
* @uuid: UUID which is booting.
* Returns integer: success (0) or error (< 0)
* @vgpu_shutdown: Called to teardown vGPU related resources for
* the VM
* @uuid: UUID which is shutting down .
* Returns integer: success (0) or error (< 0)
* @read: Read emulation callback
* @vdev: vgpu device structure
* @buf: read buffer
* @count: number bytes to read
* @address_space: specifies for which address space
* the
Re: [Qemu-devel] [RFC PATCH v1 1/1] vGPU core driver : to provide common interface for vGPU.
> From: Kirti Wankhede [mailto:kwankh...@nvidia.com] > Sent: Tuesday, February 02, 2016 9:48 AM > > Resending this mail again, somehow my previous mail didn't reached every > to everyone's inbox. > > On 2/2/2016 3:16 AM, Kirti Wankhede wrote: > > Design for vGPU Driver: > > Main purpose of vGPU driver is to provide a common interface for vGPU > > management that can be used by differnt GPU drivers. Thanks for composing this design which is a good start. > > > > This module would provide a generic interface to create the device, add > > it to vGPU bus, add device to IOMMU group and then add it to vfio group. > > > > High Level block diagram: > > > > > > +--+vgpu_register_driver()+---+ > > | __init() +->+ | > > | | | | > > | +<-+vgpu.ko| > > | vfio_vgpu.ko | probe()/remove() | | > > | |+-+ +-+ > > +--+| +---+---+ | > > | ^ | > > | callback| | > > | +---++| > > | |vgpu_register_device() | > > | ||| > > +---^-+-++-+--+-+ > > | nvidia.ko || i915.ko | > > | ||| > > +---+++ > > > > vGPU driver provides two types of registration interfaces: Looks you missed callbacks which vgpu.ko provides to vfio_vgpu.ko, e.g. to retrieve basic region/resource info, etc... Also for GPU driver interfaces, better to identify the caller. E.g. it's easy to understand life-cycle management would come from sysfs by mgmt. stack like libvirt. What about @read and @write? what's the connection between this vgpu core driver and specific hypervisor? etc. Better to connect all necessary dots so we can refine all necessary requirements on this proposal. [...] > > > > 2. GPU driver interface > > > > /** > > * struct gpu_device_ops - Structure to be registered for each physical > > GPU to > > * register the device to vgpu module. > > * > > * @owner: The module owner. > > * @vgpu_supported_config: Called to get information about supported > > * vgpu types. > > * @dev : pci device structure of physical GPU. > > * @config: should return string listing supported > > * config > > * Returns integer: success (0) or error (< 0) > > * @vgpu_create:Called to allocate basic resouces in graphics > > * driver for a particular vgpu. > > * @dev: physical pci device structure on which vgpu > > *should be created > > * @uuid: uuid for which VM it is intended to > > * @instance: vgpu instance in that VM > > * @vgpu_id: This represents the type of vgpu to be > > *created > > * Returns integer: success (0) or error (< 0) Specifically for Intel GVT-g we didn't hard partition resource among vGPUs. Instead we allow user to accurately control how many physical resources are allocated to a vGPU. So this interface should be extensible to allow vendor specific resource control. And for UUID, I remember Alex had a concern on using it in kernel. Honestly speaking I don't have a good idea here. In Xen side there is a VM ID which can be easily used as the index. But for KVM, what would be the best identifier to associate with a VM? > > * @vgpu_destroy: Called to free resources in graphics driver for > > * a vgpu instance of that VM. > > * @dev: physical pci device structure to which > > * this vgpu points to. > > * @uuid: uuid for which the vgpu belongs to. > > * @instance: vgpu instance in that VM > > * Returns integer: success (0) or error (< 0) > > * If VM is running and vgpu_destroy is called that > > * means the vGPU is being hotunpluged. Return error > > * if VM is running and graphics driver doesn't > > * support vgpu hotplug. > > * @vgpu_start: Called to do initiate vGPU initialization > > * process in graphics driver when VM boots before > > *
