RE: [PATCH 0/13 v7] PCI: Linux kernel SR-IOV support
Jesse Barnes wrote: Hm, that's not the answer I was hoping for. :) (Was looking for, Yeah we just need this bits queued and we'll send an update for e1000 right away. :) I really don't want the SR-IOV stuff to sit out another merge cycle though... Arg. We will have drivers that support these API's posted to the lists within two or three days. These drivers are RFC only and not to be pushed upstream. More non-Xen testing needs to happen with the 82576 HW. - Greg -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/13 v7] PCI: Linux kernel SR-IOV support
On Wednesday, December 17, 2008 6:15 am Matthew Wilcox wrote: On Tue, Dec 16, 2008 at 03:23:53PM -0800, Jesse Barnes wrote: I applied 1-9 to my linux-next branch; and at least patch #10 needs a respin, I still object to #2. We should have the flexibility to have 'struct resource's that are not in this array in the pci_dev. I would like to see the SR-IOV resources _not_ in this array (and indeed, I'd like to see PCI bridges keep their producer resources somewhere other than in this array). I accept that there are still some problems with this, but patch #2 moves us further from being able to achieve this goal, not closer. Yeah, I can see what you mean here... but on the other hand it makes the existing code a bit clearer (no extra args), and really it doesn't push us *that* much further from non-pci_dev tied resources. Any patches in that direction will just get a few lines bigger, that's all. But I agree that eventually we may want to have non-pci_dev resource lists, especially if we start adding advanced host bridge drivers or something. -- Jesse Barnes, Intel Open Source Technology Center -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/13 v7] PCI: Linux kernel SR-IOV support
A: No. Q: Should I include quotations after my reply? On Wed, Dec 17, 2008 at 08:44:03AM -0800, Rose, Gregory V wrote: As noted in the attached email to the netdev list, we (e1000_devel) will support the API. Great, will you have patches for the existing e1000 drivers soon to use it? Or will they be a while before they can be available? As it is, the one posted user of this api is for a driver that has been rejected, so as there are no users of the api, I feel it should be deferrred until there is a user to make sure it all works and feels proper. thanks, greg k-h -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [PATCH 0/13 v7] PCI: Linux kernel SR-IOV support
From: linux-pci-ow...@vger.kernel.org [mailto:linux-pci- ow...@vger.kernel.org] On Behalf Of Jesse Barnes Sent: 16 December 2008 23:24 To: Yu Zhao Cc: linux-...@vger.kernel.org; Chiang, Alexander; Helgaas, Bjorn; grund...@parisc-linux.org; g...@kroah.com; mi...@elte.hu; matt...@wil.cx; randy.dun...@oracle.com; rdre...@cisco.com; ho...@verge.net.au; ying...@kernel.org; linux-ker...@vger.kernel.org; kvm@vger.kernel.org; virtualizat...@lists.linux-foundation.org Subject: Re: [PATCH 0/13 v7] PCI: Linux kernel SR-IOV support On Friday, November 21, 2008 10:36 am Yu Zhao wrote: Greetings, Following patches are intended to support SR-IOV capability in the Linux kernel. With these patches, people can turn a PCI device with the capability into multiple ones from software perspective, which will benefit KVM and achieve other purposes such as QoS, security, and etc. The Physical Function and Virtual Function drivers using the SR-IOV APIs will come soon! Major changes from v6 to v7: 1, remove boot-time resource rebalancing support. (Greg KH) 2, emit uevent upon the PF driver is loaded. (Greg KH) 3, put SR-IOV callback function into the 'pci_driver'. (Matthew Wilcox) 4, register SR-IOV service at the PF loading stage. 5, remove unnecessary APIs (pci_iov_enable/disable). Thanks for your patience with this, Yu, I know it's been a long haul. :) I applied 1-9 to my linux-next branch; and at least patch #10 needs a respin, so can you re-do 10-13 as a new patch set? On re-reading the last thread, there was a lot of smoke, but very little fire afaict. The main questions I saw were: 1) do we need SR-IOV at all? why not just make each subsystem export devices to guests? This is a bit of a red herring. Nothing about SR-IOV prevents us from making subsystems more v12n friendly. And since SR-IOV is a hardware feature supported by devices these days, we should make Linux support it. 2) should the PF/VF drivers be the same or not? Again, the SR-IOV patchset and PCI spec don't dictate this. We're free to do what we want here. 3) should VF devices be represented by pci_dev structs? Yes. (This is an easy one :) 4) can VF devices be used on the host? Yet again, SR-IOV doesn't dictate this. Developers can make PF/VF combo drivers or split them, and export the resulting devices however they want. Some subsystem work may be needed to make this efficient, but SR- IOV itself is agnostic about it. So overall I didn't see many objections to the actual code in the last post, and the issues above certainly don't merit a NAK IMO... I have two minor comments on this topic. 1) Currently the PF driver is called before the kernel initializes VFs and their resources, and the current API does not allow the PF driver to detect that easily if the allocation of the VFs and their resources has succeeded or not. It would be quite useful if the PF driver gets notified when the VFs have been created successfully as it might have to do further device-specific work *after* IOV has been enabled. 2) Configuration of SR-IOV: the current API allows to enable/disable VFs from userspace via SYSFS. At the moment I am not quite clear what exactly is supposed to control these capabilities. This could be Linux tools or, on a virtualized system, hypervisor control tools. One thing I am missing though is an in-kernel API for this which I think might be useful. After all the PF driver controls the device, and, for example, when a device error occurs (e.g. a hardware failure which only the PF driver will be able to detect, not Linux), then the PF driver might have to de-allocate all resources, shut down VFs and reset the device, or something like that. In that case the PF driver needs to have a way to notify the Linux SR-IOV code about this and initiate cleaning up of VFs and their resources. At the moment, this would have to go through userspace, I believe, and I think that is not an optimal solution. Yu, do you have an opinion on how this would be realized? Anna -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/13 v7] PCI: Linux kernel SR-IOV support
On Wednesday, December 17, 2008 3:42 am Fischer, Anna wrote: I have two minor comments on this topic. 1) Currently the PF driver is called before the kernel initializes VFs and their resources, and the current API does not allow the PF driver to detect that easily if the allocation of the VFs and their resources has succeeded or not. It would be quite useful if the PF driver gets notified when the VFs have been created successfully as it might have to do further device-specific work *after* IOV has been enabled. You're thinking of after the VFs are created the VF drivers (which may or may not be part of the PF driver) may not be able to communicate back to the PF driver that something else needs to be done (I remember seeing this in the earlier thread, should have included it in my post, sorry)? I'm not sure if it makes sense to add an interface like that to the core until we have feel for what the PF/VF drivers are going to want... Or do you have something specific in mind right now? If/until we have something in the core, it seems like this could be done on a per PF/VF driver basis for now. 2) Configuration of SR-IOV: the current API allows to enable/disable VFs from userspace via SYSFS. At the moment I am not quite clear what exactly is supposed to control these capabilities. This could be Linux tools or, on a virtualized system, hypervisor control tools. One thing I am missing though is an in-kernel API for this which I think might be useful. After all the PF driver controls the device, and, for example, when a device error occurs (e.g. a hardware failure which only the PF driver will be able to detect, not Linux), then the PF driver might have to de-allocate all resources, shut down VFs and reset the device, or something like that. In that case the PF driver needs to have a way to notify the Linux SR-IOV code about this and initiate cleaning up of VFs and their resources. At the moment, this would have to go through userspace, I believe, and I think that is not an optimal solution. Yu, do you have an opinion on how this would be realized? That's a good point, Yu? -- Jesse Barnes, Intel Open Source Technology Center -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [PATCH 0/13 v7] PCI: Linux kernel SR-IOV support
-Original Message- From: Jesse Barnes [mailto:jbar...@virtuousgeek.org] On Wednesday, December 17, 2008 8:44 am Rose, Gregory V wrote: As noted in the attached email to the netdev list, we (e1000_devel) will support the API. Do you think you'll have those changes ready for 2.6.29? Would merging core SR-IOV support now make that any more likely? I'm not sure about readiness for 2.6.29. I can tell you that as soon as I get a Xen Dom0 kernel with these API's included it will take me less than a day to convert over to them from the current drivers I have that are using an older API from back in August. The drivers are mostly functional, they have a few bugs. I could do some quick regression testing to make sure that the API changes haven't broken anything and then some bug fixes to get everything ready for release. Maybe two or three weeks for the major bugs. I'll be out over the Christmas holidays so that puts us into middle or late January if I got the Xen Dom0 kernel today. That seems unlikely but it gives you an idea of the time required. - Greg -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/13 v7] PCI: Linux kernel SR-IOV support
On Wednesday, December 17, 2008 8:44 am Rose, Gregory V wrote: As noted in the attached email to the netdev list, we (e1000_devel) will support the API. Do you think you'll have those changes ready for 2.6.29? Would merging core SR-IOV support now make that any more likely? Thanks, Jesse -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [PATCH 0/13 v7] PCI: Linux kernel SR-IOV support
As noted in the attached email to the netdev list, we (e1000_devel) will support the API. - Greg Rose -Original Message- From: kvm-ow...@vger.kernel.org [mailto:kvm-ow...@vger.kernel.org] On Behalf Of Greg KH Sent: Tuesday, December 16, 2008 10:06 PM To: Jike Song Cc: Jesse Barnes; Zhao, Yu; linux-...@vger.kernel.org; achi...@hp.com; bjorn.helg...@hp.com; grund...@parisc-linux.org; mi...@elte.hu; matt...@wil.cx; randy.dun...@oracle.com; rdre...@cisco.com; ho...@verge.net.au; ying...@kernel.org; linux-ker...@vger.kernel.org; kvm@vger.kernel.org; virtualizat...@lists.linux-foundation.org Subject: Re: [PATCH 0/13 v7] PCI: Linux kernel SR-IOV support On Wed, Dec 17, 2008 at 10:37:54AM +0800, Jike Song wrote: Jesse Barnes wrote: Given a respin of 10-13 I think it's reasonable to merge this into 2.6.29, but I'd be much happier about it if we got some driver code along with it, so as not to have an unused interface sitting around for who knows how many releases. Is that reasonable? Do you know if any of the corresponding PF/VF driver bits are ready yet? Hi Jesse, Yu Zhao has posted a patch set with subject SR-IOV driver example at November 26, which illustrated the usage of SR-IOV API in Intel 82576 VF/PF drivers;-) Yes, but that driver was soundly rejected by the network driver maintainers, so I wouldn't go around showing that as your primary example of how to use this interface :) The point is valid, I don't think these apis should go into the tree without a driver or some other code using them. Otherwise they make no sense at all to have in-tree. thanks, greg k-h -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ---BeginMessage--- On Tue, Dec 2, 2008 at 1:27 AM, Yu Zhao yu.z...@intel.com wrote: SR-IOV drivers of Intel 82576 NIC are available. There are two parts of the drivers: Physical Function driver and Virtual Function driver. The PF driver is based on the IGB driver and is used to control PF to allocate hardware specific resources and interface with the SR-IOV core. The VF driver is a new NIC driver that is same as the traditional PCI device driver. It works in both the host and the guest (Xen and KVM) environment. These two drivers are testing versions and they are *only* intended to show how to use SR-IOV API. Intel 82576 NIC specification can be found at: http://download.intel.com/design/network/datashts/82576_Datasheet_v2p1.pdf [SR-IOV driver example 0/3 resend] introduction [SR-IOV driver example 1/3 resend] PF driver: hardware specific operations [SR-IOV driver example 2/3 resend] PF driver: integrate with SR-IOV core [SR-IOV driver example 3/3 resend] VF driver: an independent PCI NIC driver -- First of all, we (e1000-devel) do support the SR-IOV API. With that said, NAK on the driver changes. We were not involved in these changes and are currently working on a version of the drivers that will make them acceptable for kernel inclusion. -- Cheers, Jeff ---End Message---
Re: [PATCH 0/13 v7] PCI: Linux kernel SR-IOV support
Rose, Gregory V wrote: -Original Message- From: Jesse Barnes [mailto:jbar...@virtuousgeek.org] On Wednesday, December 17, 2008 8:44 am Rose, Gregory V wrote: As noted in the attached email to the netdev list, we (e1000_devel) will support the API. Do you think you'll have those changes ready for 2.6.29? Would merging core SR-IOV support now make that any more likely? I'm not sure about readiness for 2.6.29. I can tell you that as soon as I get a Xen Dom0 kernel with these API's included it will take me less than a day to convert over to them from the current drivers I have that are using an older API from back in August. Which dom0 kernel are you using? Is it based on my pvops-based dom0 work? J -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [PATCH 0/13 v7] PCI: Linux kernel SR-IOV support
Jeremy Fitzhardinge wrote: Which dom0 kernel are you using? Is it based on my pvops-based dom0 work? The kernel I'm currently using is an ad-hoc patchwork of changes to the 2.6.18 Xen Dom0 kernel that was available back in August. The folks from OTC in Intel (Zhao Yu and his team) would be able to provide you more background on it as they did the work to enable MSI-X, SR-IOV and VT-d in that kernel so that my drivers would function. I don't see Zhao Yu on the distro list for this email so I'll add him. - Greg -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/13 v7] PCI: Linux kernel SR-IOV support
On Wednesday, December 17, 2008 11:05 am Rose, Gregory V wrote: -Original Message- From: Jesse Barnes [mailto:jbar...@virtuousgeek.org] On Wednesday, December 17, 2008 8:44 am Rose, Gregory V wrote: As noted in the attached email to the netdev list, we (e1000_devel) will support the API. Do you think you'll have those changes ready for 2.6.29? Would merging core SR-IOV support now make that any more likely? I'm not sure about readiness for 2.6.29. I can tell you that as soon as I get a Xen Dom0 kernel with these API's included it will take me less than a day to convert over to them from the current drivers I have that are using an older API from back in August. The drivers are mostly functional, they have a few bugs. I could do some quick regression testing to make sure that the API changes haven't broken anything and then some bug fixes to get everything ready for release. Maybe two or three weeks for the major bugs. I'll be out over the Christmas holidays so that puts us into middle or late January if I got the Xen Dom0 kernel today. That seems unlikely but it gives you an idea of the time required. Hm, that's not the answer I was hoping for. :) (Was looking for, Yeah we just need this bits queued and we'll send an update for e1000 right away. :) I really don't want the SR-IOV stuff to sit out another merge cycle though... Arg. -- Jesse Barnes, Intel Open Source Technology Center -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/13 v7] PCI: Linux kernel SR-IOV support
On Tue, Dec 16, 2008 at 03:23:53PM -0800, Jesse Barnes wrote: I applied 1-9 to my linux-next branch; and at least patch #10 needs a respin, I still object to #2. We should have the flexibility to have 'struct resource's that are not in this array in the pci_dev. I would like to see the SR-IOV resources _not_ in this array (and indeed, I'd like to see PCI bridges keep their producer resources somewhere other than in this array). I accept that there are still some problems with this, but patch #2 moves us further from being able to achieve this goal, not closer. -- Matthew Wilcox Intel Open Source Technology Centre Bill, look, we understand that you're interested in selling us this operating system, but compare it to ours. We can't possibly take such a retrograde step. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/13 v7] PCI: Linux kernel SR-IOV support
On Wednesday, December 17, 2008 11:51 am Greg KH wrote: On Wed, Dec 17, 2008 at 11:42:54AM -0800, Jesse Barnes wrote: I really don't want the SR-IOV stuff to sit out another merge cycle though... Arg. Why, is there some rush to get it in? As there is no in-kernel users of it, I don't see the problem with postponing it until someone actually needs it. Well it *does* make development of SR-IOV drivers that much harder. As you know, out of tree development is a pain. OTOH if any changes end up being required, they can be done before the code is merged. Anyway, hopefully we won't have to worry about it because some driver will come along soon that uses Yu's code. :) If not, Yu might have to maintain a separate git tree or something until the drivers are ready to be merged. -- Jesse Barnes, Intel Open Source Technology Center -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/13 v7] PCI: Linux kernel SR-IOV support
Fischer, Anna wrote: I have two minor comments on this topic. 1) Currently the PF driver is called before the kernel initializes VFs and their resources, and the current API does not allow the PF driver to detect that easily if the allocation of the VFs and their resources has succeeded or not. It would be quite useful if the PF driver gets notified when the VFs have been created successfully as it might have to do further device-specific work *after* IOV has been enabled. If the VF allocation fails in the PCI layer, then the SR-IOV core will invokes the callback again to notify the PF driver with zero VF count. The PF driver does not have to concern about this even the PCI layer code fails (and actually it's very rare). And I'm not sure why the PF driver wants to do further work *after* the VF is allocated. Does this mean PF driver have to set up some internal resources related to SR-IOV/VF? If yes, I suggest the PF driver do it before VF allocation. The design philosophy of SR-IOV/VF is that VF is treated as hot-plug device, which means it should be immediately usable by VF driver (e.g. VF driver is pre-loaded) after it appears in the PCI subsystem. If that is not the purpose, then PF driver should handle it not depending on the SR-IOV, right? If you could elaborate your SR-IOV PF/VF h/w specific requirement, it would be help for me to answer this question :-) 2) Configuration of SR-IOV: the current API allows to enable/disable VFs from userspace via SYSFS. At the moment I am not quite clear what exactly is supposed to control these capabilities. This could be Linux tools or, on a virtualized system, hypervisor control tools. This depends on user application, you know, which depends on the usage environment (i.e. native, KVM or Xen). One thing I am missing though is an in-kernel API for this which I think might be useful. After all the PF driver controls the device, and, for example, when a device error occurs (e.g. a hardware failure which only the PF driver will be able to detect, not Linux), then the PF driver might have to de-allocate all resources, shut down VFs and reset the device, or something like that. In that case the PF driver needs to have a way to notify the Linux SR-IOV code about this and initiate cleaning up of VFs and their resources. At the moment, this would have to go through userspace, I believe, and I think that is not an optimal solution. Yu, do you have an opinion on how this would be realized? Yes, the PF driver can use pci_iov_unregister to disable SR-IOV in case the fatal error occurs. This function also sends notification to user level through 'uevent' so user application can aware the change. Thanks, Yu -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/13 v7] PCI: Linux kernel SR-IOV support
Matthew Wilcox wrote: On Tue, Dec 16, 2008 at 03:23:53PM -0800, Jesse Barnes wrote: I applied 1-9 to my linux-next branch; and at least patch #10 needs a respin, I still object to #2. We should have the flexibility to have 'struct resource's that are not in this array in the pci_dev. I would like to see the SR-IOV resources _not_ in this array (and indeed, I'd like to see PCI bridges keep their producer resources somewhere other than in this array). I accept that there are still some problems with this, but I understand your concern, and agree that using the array as resource manager is not the best way. But for now it's not possible as you know. We need a better resource manager for PCI subsystem to manage the various resources (traditional, device specific, bus related), which is another independent work from SR-IOV change. patch #2 moves us further from being able to achieve this goal, not closer. The array is obvious straightforward and can be easily replaced with a more advanced resource manager in the future. So I don't think we going further from or closer to the goal. Thanks, Yu -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/13 v7] PCI: Linux kernel SR-IOV support
Jesse Barnes wrote: On Wednesday, December 17, 2008 11:51 am Greg KH wrote: On Wed, Dec 17, 2008 at 11:42:54AM -0800, Jesse Barnes wrote: I really don't want the SR-IOV stuff to sit out another merge cycle though... Arg. Why, is there some rush to get it in? As there is no in-kernel users of it, I don't see the problem with postponing it until someone actually needs it. Well it *does* make development of SR-IOV drivers that much harder. As you know, out of tree development is a pain. OTOH if any changes end up being required, they can be done before the code is merged. Yes, people write to me asking for the SR-IOV patch or update everyday -- I guess they don't want to let their competitors know they are working on it so they can't bring their questions up on the mailing list. And I personally also have dozen of other patches related to PCI and KVM subsystems which depend on the SR-IOV change. Anyway, hopefully we won't have to worry about it because some driver will come along soon that uses Yu's code. :) If not, Yu might have to maintain a separate git tree or something until the drivers are ready to be merged. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [PATCH 0/13 v7] PCI: Linux kernel SR-IOV support
From: Zhao, Yu [mailto:yu.z...@intel.com] Sent: 18 December 2008 02:14 To: Fischer, Anna Cc: Jesse Barnes; linux-...@vger.kernel.org; Chiang, Alexander; Helgaas, Bjorn; grund...@parisc-linux.org; g...@kroah.com; mi...@elte.hu; matt...@wil.cx; randy.dun...@oracle.com; rdre...@cisco.com; ho...@verge.net.au; ying...@kernel.org; linux- ker...@vger.kernel.org; kvm@vger.kernel.org; virtualizat...@lists.linux-foundation.org Subject: Re: [PATCH 0/13 v7] PCI: Linux kernel SR-IOV support Fischer, Anna wrote: I have two minor comments on this topic. 1) Currently the PF driver is called before the kernel initializes VFs and their resources, and the current API does not allow the PF driver to detect that easily if the allocation of the VFs and their resources has succeeded or not. It would be quite useful if the PF driver gets notified when the VFs have been created successfully as it might have to do further device-specific work *after* IOV has been enabled. If the VF allocation fails in the PCI layer, then the SR-IOV core will invokes the callback again to notify the PF driver with zero VF count. The PF driver does not have to concern about this even the PCI layer code fails (and actually it's very rare). Yes, this is good. And I'm not sure why the PF driver wants to do further work *after* the VF is allocated. Does this mean PF driver have to set up some internal resources related to SR-IOV/VF? If yes, I suggest the PF driver do it before VF allocation. The design philosophy of SR-IOV/VF is that VF is treated as hot-plug device, which means it should be immediately usable by VF driver (e.g. VF driver is pre-loaded) after it appears in the PCI subsystem. If that is not the purpose, then PF driver should handle it not depending on the SR-IOV, right? Yes, you are right. In fact I was assuming in this case that the PF driver might have to allocate VF specific resources before a PF - VF communication can be established but this can be done before the VF PCI device appears, so I was wrong with this. The current API is sufficient to handle all of this, so I am withdrawing my concern here ;-) If you could elaborate your SR-IOV PF/VF h/w specific requirement, it would be help for me to answer this question :-) 2) Configuration of SR-IOV: the current API allows to enable/disable VFs from userspace via SYSFS. At the moment I am not quite clear what exactly is supposed to control these capabilities. This could be Linux tools or, on a virtualized system, hypervisor control tools. This depends on user application, you know, which depends on the usage environment (i.e. native, KVM or Xen). One thing I am missing though is an in-kernel API for this which I think might be useful. After all the PF driver controls the device, and, for example, when a device error occurs (e.g. a hardware failure which only the PF driver will be able to detect, not Linux), then the PF driver might have to de-allocate all resources, shut down VFs and reset the device, or something like that. In that case the PF driver needs to have a way to notify the Linux SR-IOV code about this and initiate cleaning up of VFs and their resources. At the moment, this would have to go through userspace, I believe, and I think that is not an optimal solution. Yu, do you have an opinion on how this would be realized? Yes, the PF driver can use pci_iov_unregister to disable SR-IOV in case the fatal error occurs. This function also sends notification to user level through 'uevent' so user application can aware the change. If pci_iov_unregister is accessible for kernel drivers than this is in fact all we need. Thanks for the clarification. I think the patchset looks very good. Acked-by: Anna Fischer anna.fisc...@hp.com -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/13 v7] PCI: Linux kernel SR-IOV support
On Friday, November 21, 2008 10:36 am Yu Zhao wrote: Greetings, Following patches are intended to support SR-IOV capability in the Linux kernel. With these patches, people can turn a PCI device with the capability into multiple ones from software perspective, which will benefit KVM and achieve other purposes such as QoS, security, and etc. The Physical Function and Virtual Function drivers using the SR-IOV APIs will come soon! Major changes from v6 to v7: 1, remove boot-time resource rebalancing support. (Greg KH) 2, emit uevent upon the PF driver is loaded. (Greg KH) 3, put SR-IOV callback function into the 'pci_driver'. (Matthew Wilcox) 4, register SR-IOV service at the PF loading stage. 5, remove unnecessary APIs (pci_iov_enable/disable). Thanks for your patience with this, Yu, I know it's been a long haul. :) I applied 1-9 to my linux-next branch; and at least patch #10 needs a respin, so can you re-do 10-13 as a new patch set? On re-reading the last thread, there was a lot of smoke, but very little fire afaict. The main questions I saw were: 1) do we need SR-IOV at all? why not just make each subsystem export devices to guests? This is a bit of a red herring. Nothing about SR-IOV prevents us from making subsystems more v12n friendly. And since SR-IOV is a hardware feature supported by devices these days, we should make Linux support it. 2) should the PF/VF drivers be the same or not? Again, the SR-IOV patchset and PCI spec don't dictate this. We're free to do what we want here. 3) should VF devices be represented by pci_dev structs? Yes. (This is an easy one :) 4) can VF devices be used on the host? Yet again, SR-IOV doesn't dictate this. Developers can make PF/VF combo drivers or split them, and export the resulting devices however they want. Some subsystem work may be needed to make this efficient, but SR-IOV itself is agnostic about it. So overall I didn't see many objections to the actual code in the last post, and the issues above certainly don't merit a NAK IMO... Given a respin of 10-13 I think it's reasonable to merge this into 2.6.29, but I'd be much happier about it if we got some driver code along with it, so as not to have an unused interface sitting around for who knows how many releases. Is that reasonable? Do you know if any of the corresponding PF/VF driver bits are ready yet? Thanks, -- Jesse Barnes, Intel Open Source Technology Center -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/13 v7] PCI: Linux kernel SR-IOV support
Jesse Barnes wrote: Given a respin of 10-13 I think it's reasonable to merge this into 2.6.29, but I'd be much happier about it if we got some driver code along with it, so as not to have an unused interface sitting around for who knows how many releases. Is that reasonable? Do you know if any of the corresponding PF/VF driver bits are ready yet? Hi Jesse, Yu Zhao has posted a patch set with subject SR-IOV driver example at November 26, which illustrated the usage of SR-IOV API in Intel 82576 VF/PF drivers;-) -- Thanks, Jike -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/13 v7] PCI: Linux kernel SR-IOV support
On Wed, Dec 17, 2008 at 10:37:54AM +0800, Jike Song wrote: Jesse Barnes wrote: Given a respin of 10-13 I think it's reasonable to merge this into 2.6.29, but I'd be much happier about it if we got some driver code along with it, so as not to have an unused interface sitting around for who knows how many releases. Is that reasonable? Do you know if any of the corresponding PF/VF driver bits are ready yet? Hi Jesse, Yu Zhao has posted a patch set with subject SR-IOV driver example at November 26, which illustrated the usage of SR-IOV API in Intel 82576 VF/PF drivers;-) Yes, but that driver was soundly rejected by the network driver maintainers, so I wouldn't go around showing that as your primary example of how to use this interface :) The point is valid, I don't think these apis should go into the tree without a driver or some other code using them. Otherwise they make no sense at all to have in-tree. thanks, greg k-h -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/13 v7] PCI: Linux kernel SR-IOV support
Greg KH wrote: On Wed, Dec 17, 2008 at 10:37:54AM +0800, Jike Song wrote: Jesse Barnes wrote: Given a respin of 10-13 I think it's reasonable to merge this into 2.6.29, but I'd be much happier about it if we got some driver code along with it, so as not to have an unused interface sitting around for who knows how many releases. Is that reasonable? Do you know if any of the corresponding PF/VF driver bits are ready yet? Hi Jesse, Yu Zhao has posted a patch set with subject SR-IOV driver example at November 26, which illustrated the usage of SR-IOV API in Intel 82576 VF/PF drivers;-) Yes, but that driver was soundly rejected by the network driver maintainers, so I wouldn't go around showing that as your primary example of how to use this interface :) The point is valid, I don't think these apis should go into the tree without a driver or some other code using them. Otherwise they make no sense at all to have in-tree. I agree the point is valid, but on another hand this is a 'the chicken the egg' problem -- if we don't have the SR-IOV base, people who are developing PF drivers can not get their changes in-tree. Maybe they are holding the patches and waiting on the infrastructure... :-) -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/13 v7] PCI: Linux kernel SR-IOV support
On Wed, Dec 17, 2008 at 03:07:23PM +0800, Zhao, Yu wrote: Greg KH wrote: On Wed, Dec 17, 2008 at 10:37:54AM +0800, Jike Song wrote: Jesse Barnes wrote: Given a respin of 10-13 I think it's reasonable to merge this into 2.6.29, but I'd be much happier about it if we got some driver code along with it, so as not to have an unused interface sitting around for who knows how many releases. Is that reasonable? Do you know if any of the corresponding PF/VF driver bits are ready yet? Hi Jesse, Yu Zhao has posted a patch set with subject SR-IOV driver example at November 26, which illustrated the usage of SR-IOV API in Intel 82576 VF/PF drivers;-) Yes, but that driver was soundly rejected by the network driver maintainers, so I wouldn't go around showing that as your primary example of how to use this interface :) The point is valid, I don't think these apis should go into the tree without a driver or some other code using them. Otherwise they make no sense at all to have in-tree. I agree the point is valid, but on another hand this is a 'the chicken the egg' problem -- if we don't have the SR-IOV base, people who are developing PF drivers can not get their changes in-tree. Maybe they are holding the patches and waiting on the infrastructure... :-) Are they? They can both go in at the same time, like almost every other api addition to the kernel, right? thanks, greg k-h -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/13 v7] PCI: Linux kernel SR-IOV support
On Sat, Nov 22, 2008 at 02:36:05AM +0800, Yu Zhao wrote: Greetings, Following patches are intended to support SR-IOV capability in the Linux kernel. With these patches, people can turn a PCI device with the capability into multiple ones from software perspective, which will benefit KVM and achieve other purposes such as QoS, security, and etc. The Physical Function and Virtual Function drivers using the SR-IOV APIs will come soon! Thanks for respining these patches, but I think we really need to see a driver using this in order to get an idea of how it will be used. Also, the Xen and KVM people need to agree on the userspace interface here, perhaps also getting some libvirt involvement as well, as they are going to be the ones having to use this all the time. thanks, greg k-h -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/13 v7] PCI: Linux kernel SR-IOV support
Greg KH wrote: On Sat, Nov 22, 2008 at 02:36:05AM +0800, Yu Zhao wrote: Greetings, Following patches are intended to support SR-IOV capability in the Linux kernel. With these patches, people can turn a PCI device with the capability into multiple ones from software perspective, which will benefit KVM and achieve other purposes such as QoS, security, and etc. The Physical Function and Virtual Function drivers using the SR-IOV APIs will come soon! Thanks for respining these patches, but I think we really need to see a driver using this in order to get an idea of how it will be used. Yes, the PF driver patch and the VF driver for Intel 82576 NIC will be available next week. Both PF/VF drivers are testing versions. The PF driver patch is based on the lasted kernel IGB driver (drivers/net/igb/), and uses SR-IOV v7 API. The VF driver is a totally new NIC driver, it looks like other normal PCI NIC drivers. Also, the Xen and KVM people need to agree on the userspace interface here, perhaps also getting some libvirt involvement as well, as they are going to be the ones having to use this all the time. I'll keep KVM/Xen people updated on the user level interface and let them comment based on a real usage model of the Intel 82576 NIC after PF/VF drivers are available. Regards, Yu -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
