Re: [PATCH V14 00/21] Enable SRIOV on Power8
2015-03-21 5:20 GMT+08:00 Bjorn Helgaas : > On Fri, Mar 20, 2015 at 11:06:16AM +0800, Wei Yang wrote: >> This patchset enables the SRIOV on POWER8. >> >> The general idea is put each VF into one individual PE and allocate required >> resources like MMIO/DMA/MSI. The major difficulty comes from the MMIO >> allocation and adjustment for PF's IOV BAR. >> >> On P8, we use M64BT to cover a PF's IOV BAR, which could make an individual >> VF >> sit in its own PE. This gives more flexiblity, while at the mean time it >> brings on some restrictions on the PF's IOV BAR size and alignment. > > For patches 03-12 (the "PCI: " ones): > > Acked-by: Bjorn Helgaas > > It's fine if you want to take all these via the powerpc tree. I don't see > much if anything in my queue that will conflict with them. > Thanks, Bjorn :-) I will work with Ben/Gavin to merge it on ppc. If anything change in PCI core, I will let you know. Have a nice weekend :-) > Bjorn > ___ > Linuxppc-dev mailing list > Linuxppc-dev@lists.ozlabs.org > https://lists.ozlabs.org/listinfo/linuxppc-dev -- Richard Yang Help You, Help Me ___ Linuxppc-dev mailing list Linuxppc-dev@lists.ozlabs.org https://lists.ozlabs.org/listinfo/linuxppc-dev
Re: [PATCH V14 00/21] Enable SRIOV on Power8
On Fri, 2015-03-20 at 16:16 -0500, Bjorn Helgaas wrote: > > > The other option is you put the generic changes in a topic branch > which > > both of us merge and I take care of the rest. > > Yeah, I chatted with Wei this morning, and he reminded me that there's > really not very much drivers/pci content there. So I think I can just > ack those bits and you can take the whole thing. I noticed some > changelog typos I'll point out, then I'll be done. Excellent, thanks. Cheers, Ben. ___ Linuxppc-dev mailing list Linuxppc-dev@lists.ozlabs.org https://lists.ozlabs.org/listinfo/linuxppc-dev
Re: [PATCH V14 00/21] Enable SRIOV on Power8
On Fri, Mar 20, 2015 at 11:06:16AM +0800, Wei Yang wrote: > This patchset enables the SRIOV on POWER8. > > The general idea is put each VF into one individual PE and allocate required > resources like MMIO/DMA/MSI. The major difficulty comes from the MMIO > allocation and adjustment for PF's IOV BAR. > > On P8, we use M64BT to cover a PF's IOV BAR, which could make an individual VF > sit in its own PE. This gives more flexiblity, while at the mean time it > brings on some restrictions on the PF's IOV BAR size and alignment. For patches 03-12 (the "PCI: " ones): Acked-by: Bjorn Helgaas It's fine if you want to take all these via the powerpc tree. I don't see much if anything in my queue that will conflict with them. Bjorn ___ Linuxppc-dev mailing list Linuxppc-dev@lists.ozlabs.org https://lists.ozlabs.org/listinfo/linuxppc-dev
Re: [PATCH V14 00/21] Enable SRIOV on Power8
On Fri, Mar 20, 2015 at 4:05 PM, Benjamin Herrenschmidt wrote: > On Fri, 2015-03-20 at 09:34 -0500, Bjorn Helgaas wrote: >> On Fri, Mar 20, 2015 at 12:43 AM, Gavin Shan >> wrote: >> > On Fri, Mar 20, 2015 at 11:06:16AM +0800, Wei Yang wrote: >> > >> > [snip] >> > >> >>--- >> >>v14: >> >> * call ppc_md.pcibios_fixup_sriov() in pcibios_add_device >> >> * add more explanation in change log >> >> * Following patches have been reordered to the beginning. >> >> "EEH refactor to use pci_dn": >> >> 8ec20d6 powerpc/powernv: Use pci_dn, not device_node, in PCI config >> >> accessor >> >> a3460fc powerpc/pci: Refactor pci_dn >> >> These two patches will be modified to merge with other patches which >> >> are >> >> under discussion/review in ppc mail list. Some changes may also be >> >> made in >> >> other patches, which I didn't include them in this series, so that the >> >> auto build robot could work on this. >> > >> > The comment here isn't precise enough and not the things I suggested >> > before. >> > Those 2 patches have been split into 3 patches (A/B/C). Some other EEH >> > cleanup/refactor patches depends on A/B and those patches would be merged >> > before your SRIOV patches to PowerPC tree. C, which I already sent to you, >> > need to be integrated to your patchset right after the following one: >> > >> > powerpc/pci: Don't unset PCI resources for VFs >> > >> > I guess you can move the patches around after checking if Bjorn has further >> > concerns/comments. >> >> I think we need to move forward on this soon so it has at least a >> little time in -next. And it sounds like I cannot merge this v14 >> as-is because it doesn't include these additional EEH cleanup/refactor >> patches? (If they are only cleanup and refactoring, I don't >> understand why they would be required, but I haven't seen them, so I'm >> taking your word for it.) >> >> So I guess I'm waiting for v15. Don't wait for my comments on v14. >> Ideally I'd also like an ack from a powerpc maintainer. This has so >> much powerpc impact that I'm not comfortable stuffing this in without >> their OK. > > The other option is you put the generic changes in a topic branch which > both of us merge and I take care of the rest. Yeah, I chatted with Wei this morning, and he reminded me that there's really not very much drivers/pci content there. So I think I can just ack those bits and you can take the whole thing. I noticed some changelog typos I'll point out, then I'll be done. Bjorn ___ Linuxppc-dev mailing list Linuxppc-dev@lists.ozlabs.org https://lists.ozlabs.org/listinfo/linuxppc-dev
Re: [PATCH V14 00/21] Enable SRIOV on Power8
On Fri, 2015-03-20 at 09:34 -0500, Bjorn Helgaas wrote: > On Fri, Mar 20, 2015 at 12:43 AM, Gavin Shan > wrote: > > On Fri, Mar 20, 2015 at 11:06:16AM +0800, Wei Yang wrote: > > > > [snip] > > > >>--- > >>v14: > >> * call ppc_md.pcibios_fixup_sriov() in pcibios_add_device > >> * add more explanation in change log > >> * Following patches have been reordered to the beginning. > >> "EEH refactor to use pci_dn": > >> 8ec20d6 powerpc/powernv: Use pci_dn, not device_node, in PCI config > >> accessor > >> a3460fc powerpc/pci: Refactor pci_dn > >> These two patches will be modified to merge with other patches which > >> are > >> under discussion/review in ppc mail list. Some changes may also be > >> made in > >> other patches, which I didn't include them in this series, so that the > >> auto build robot could work on this. > > > > The comment here isn't precise enough and not the things I suggested before. > > Those 2 patches have been split into 3 patches (A/B/C). Some other EEH > > cleanup/refactor patches depends on A/B and those patches would be merged > > before your SRIOV patches to PowerPC tree. C, which I already sent to you, > > need to be integrated to your patchset right after the following one: > > > > powerpc/pci: Don't unset PCI resources for VFs > > > > I guess you can move the patches around after checking if Bjorn has further > > concerns/comments. > > I think we need to move forward on this soon so it has at least a > little time in -next. And it sounds like I cannot merge this v14 > as-is because it doesn't include these additional EEH cleanup/refactor > patches? (If they are only cleanup and refactoring, I don't > understand why they would be required, but I haven't seen them, so I'm > taking your word for it.) > > So I guess I'm waiting for v15. Don't wait for my comments on v14. > Ideally I'd also like an ack from a powerpc maintainer. This has so > much powerpc impact that I'm not comfortable stuffing this in without > their OK. The other option is you put the generic changes in a topic branch which both of us merge and I take care of the rest. Cheers, Ben. > Bjorn ___ Linuxppc-dev mailing list Linuxppc-dev@lists.ozlabs.org https://lists.ozlabs.org/listinfo/linuxppc-dev
Re: [PATCH V14 00/21] Enable SRIOV on Power8
On Fri, Mar 20, 2015 at 12:43 AM, Gavin Shan wrote: > On Fri, Mar 20, 2015 at 11:06:16AM +0800, Wei Yang wrote: > > [snip] > >>--- >>v14: >> * call ppc_md.pcibios_fixup_sriov() in pcibios_add_device >> * add more explanation in change log >> * Following patches have been reordered to the beginning. >> "EEH refactor to use pci_dn": >> 8ec20d6 powerpc/powernv: Use pci_dn, not device_node, in PCI config >> accessor >> a3460fc powerpc/pci: Refactor pci_dn >> These two patches will be modified to merge with other patches which are >> under discussion/review in ppc mail list. Some changes may also be made >> in >> other patches, which I didn't include them in this series, so that the >> auto build robot could work on this. > > The comment here isn't precise enough and not the things I suggested before. > Those 2 patches have been split into 3 patches (A/B/C). Some other EEH > cleanup/refactor patches depends on A/B and those patches would be merged > before your SRIOV patches to PowerPC tree. C, which I already sent to you, > need to be integrated to your patchset right after the following one: > > powerpc/pci: Don't unset PCI resources for VFs > > I guess you can move the patches around after checking if Bjorn has further > concerns/comments. I think we need to move forward on this soon so it has at least a little time in -next. And it sounds like I cannot merge this v14 as-is because it doesn't include these additional EEH cleanup/refactor patches? (If they are only cleanup and refactoring, I don't understand why they would be required, but I haven't seen them, so I'm taking your word for it.) So I guess I'm waiting for v15. Don't wait for my comments on v14. Ideally I'd also like an ack from a powerpc maintainer. This has so much powerpc impact that I'm not comfortable stuffing this in without their OK. Bjorn ___ Linuxppc-dev mailing list Linuxppc-dev@lists.ozlabs.org https://lists.ozlabs.org/listinfo/linuxppc-dev
Re: [PATCH V14 00/21] Enable SRIOV on Power8
On Fri, Mar 20, 2015 at 11:06:16AM +0800, Wei Yang wrote: [snip] >--- >v14: > * call ppc_md.pcibios_fixup_sriov() in pcibios_add_device > * add more explanation in change log > * Following patches have been reordered to the beginning. > "EEH refactor to use pci_dn": > 8ec20d6 powerpc/powernv: Use pci_dn, not device_node, in PCI config > accessor > a3460fc powerpc/pci: Refactor pci_dn > These two patches will be modified to merge with other patches which are > under discussion/review in ppc mail list. Some changes may also be made in > other patches, which I didn't include them in this series, so that the > auto build robot could work on this. The comment here isn't precise enough and not the things I suggested before. Those 2 patches have been split into 3 patches (A/B/C). Some other EEH cleanup/refactor patches depends on A/B and those patches would be merged before your SRIOV patches to PowerPC tree. C, which I already sent to you, need to be integrated to your patchset right after the following one: powerpc/pci: Don't unset PCI resources for VFs I guess you can move the patches around after checking if Bjorn has further concerns/comments. Thanks, Gavin > There may have several changes in powerpc arch, which not effect the pci > core. So after this patch set pass the review in pci community, I would > rebase this series on ppc brach and send out for comment. > * use add_res->min_align as the alignment in reassign_resources_sorted() > * some cleanup in Document >v13: > * fix error in pcibios_iov_resource_alignment(), use pdev instead of dev > * rename vf_num to num_vfs in pcibios_sriov_enable(), > pnv_pci_vf_resource_shift(), pnv_pci_sriov_disable(), > pnv_pci_sriov_enable(), pnv_pci_ioda2_setup_dma_pe() > * add more explanation in commit "powerpc/pci: Don't unset PCI resources > for VFs" > * fix IOV BAR in hotplug path as well, and don't fixup an already added > device > * use roundup_pow_of_two() instead of __roundup_pow_of_two() > * this is based on v4.0-rc1 >v12: > * remove "align" parameter from pcibios_iov_resource_alignment() > default version returns pci_iov_resource_size() instead of the > "align" parameter > * in powerpc pcibios_iov_resource_alignment(), return > pci_iov_resource_size() if there's no ppc_md function pointer > * in pci_sriov_resource_alignment(), don't re-read base, since we > saved the required alignment when reading it the first time > * remove "vf_num" parameter from add_dev_pci_info() and > remove_dev_pci_info(); use pci_sriov_get_totalvfs() instead > * use dev_warn() instead of pr_warn() when possible > * check to be sure IOV BAR is still in range after shifting, change > pnv_pci_vf_resource_shift() from void to int > * improve sriov_enable() error message > * improve SR-IOV BAR sizing message > * index IOV resources in conventional style > * include preamble patches (refresh offset/stride when updating numVFs, > calculate max buses required > * restructure pci_iov_max_bus_range() to return value instead of updating > internally, rename to virtfn_max_buses() > * fix typos & formatting > * expand documentation >v11: > * fix some compile warning >v10: > * remove weak function pcibios_iov_resource_size() > the VF BAR size is stored in pci_sriov structure and retrieved from > pci_iov_resource_size() > * Use "Reserve additional" instead of "Expand" to be more acurate in the > change log > * add log message to show the PF's IOV BAR final size > * add pcibios_sriov_enable/disable() weak funcion in sriov_enable/disable() > for arch setup before enable VFs. Like the arch could fix up the BDF for > VFs, since the change of NumVFs would affect the BDF of VFs. > * Add some explanation of PE on Power arch in the documentation >v9: > * make the change log consistent in the terminology > PF's IOV BAR -> the SRIOV BAR in PF > VF's BAR -> the normal BAR in VF's view > * rename all newly introduced function from _sriov_ to _iov_ > * rename the document to > Documentation/powerpc/pci_iov_resource_on_powernv.txt > * add the vendor id and device id of the tested devices > * change return value from EINVAL to ENOSYS for pci_iov_virtfn_bus() and > pci_iov_virtfn_devfn() when it is called on PF or SRIOV is not configured > * rebase on 3.18-rc2 and tested >v8: > * use weak funcion pcibios_sriov_resource_size() instead of some flag to > retrieve the IOV BAR size. > * add a document Documentation/powerpc/pci_resource.txt to explain the > design. > * make pci_iov_virtfn_bus()/pci_iov_virtfn_devfn() not inline. > * extract a function res_to_dev_res(), so that it is more general to get > additional size and alignment > * fix one contention which is introduced in "powrepc/pci: Refactor pci_dn". > the root cause is pci_get_slot() takes pci_bus_sem and leads to dead > lock. >v7: >
[PATCH V14 00/21] Enable SRIOV on Power8
This patchset enables the SRIOV on POWER8. The general idea is put each VF into one individual PE and allocate required resources like MMIO/DMA/MSI. The major difficulty comes from the MMIO allocation and adjustment for PF's IOV BAR. On P8, we use M64BT to cover a PF's IOV BAR, which could make an individual VF sit in its own PE. This gives more flexiblity, while at the mean time it brings on some restrictions on the PF's IOV BAR size and alignment. To achieve this effect, we need to do some hack on pci devices's resources. 1. Expand the IOV BAR properly. Done by pnv_pci_ioda_fixup_iov_resources(). 2. Shift the IOV BAR properly. Done by pnv_pci_vf_resource_shift(). 3. IOV BAR alignment is calculated by arch dependent function instead of an individual VF BAR size. Done by pnv_pcibios_sriov_resource_alignment(). 4. Take the IOV BAR alignment into consideration in the sizing and assigning. This is achieved by commit: "PCI: Take additional IOV BAR alignment in sizing and assigning" Test Environment: The SRIOV device tested is Emulex Lancer(10df:e220) and Mellanox ConnectX-3(15b3:1003) on POWER8. Examples on pass through a VF to guest through vfio: 1. unbind the original driver and bind to vfio-pci driver echo :06:0d.0 > /sys/bus/pci/devices/:06:0d.0/driver/unbind echo 1102 0002 > /sys/bus/pci/drivers/vfio-pci/new_id Note: this should be done for each device in the same iommu_group 2. Start qemu and pass device through vfio /home/ywywyang/git/qemu-impreza/ppc64-softmmu/qemu-system-ppc64 \ -M pseries -m 2048 -enable-kvm -nographic \ -drive file=/home/ywywyang/kvm/fc19.img \ -monitor telnet:localhost:5435,server,nowait -boot cd \ -device "spapr-pci-vfio-host-bridge,id=CXGB3,iommu=26,index=6" Verify this is the exact VF response: 1. ping from a machine in the same subnet(the broadcast domain) 2. run arp -n on this machine 9.115.251.20 ether 00:00:c9:df:ed:bf C eth0 3. ifconfig in the guest # ifconfig eth1 eth1: flags=4163 mtu 1500 inet 9.115.251.20 netmask 255.255.255.0 broadcast 9.115.251.255 inet6 fe80::200:c9ff:fedf:edbf prefixlen 64 scopeid 0x20 ether 00:00:c9:df:ed:bf txqueuelen 1000 (Ethernet) RX packets 175 bytes 13278 (12.9 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 58 bytes 9276 (9.0 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 4. They have the same MAC address Note: make sure you shutdown other network interfaces in guest. --- v14: * call ppc_md.pcibios_fixup_sriov() in pcibios_add_device * add more explanation in change log * Following patches have been reordered to the beginning. "EEH refactor to use pci_dn": 8ec20d6 powerpc/powernv: Use pci_dn, not device_node, in PCI config accessor a3460fc powerpc/pci: Refactor pci_dn These two patches will be modified to merge with other patches which are under discussion/review in ppc mail list. Some changes may also be made in other patches, which I didn't include them in this series, so that the auto build robot could work on this. There may have several changes in powerpc arch, which not effect the pci core. So after this patch set pass the review in pci community, I would rebase this series on ppc brach and send out for comment. * use add_res->min_align as the alignment in reassign_resources_sorted() * some cleanup in Document v13: * fix error in pcibios_iov_resource_alignment(), use pdev instead of dev * rename vf_num to num_vfs in pcibios_sriov_enable(), pnv_pci_vf_resource_shift(), pnv_pci_sriov_disable(), pnv_pci_sriov_enable(), pnv_pci_ioda2_setup_dma_pe() * add more explanation in commit "powerpc/pci: Don't unset PCI resources for VFs" * fix IOV BAR in hotplug path as well, and don't fixup an already added device * use roundup_pow_of_two() instead of __roundup_pow_of_two() * this is based on v4.0-rc1 v12: * remove "align" parameter from pcibios_iov_resource_alignment() default version returns pci_iov_resource_size() instead of the "align" parameter * in powerpc pcibios_iov_resource_alignment(), return pci_iov_resource_size() if there's no ppc_md function pointer * in pci_sriov_resource_alignment(), don't re-read base, since we saved the required alignment when reading it the first time * remove "vf_num" parameter from add_dev_pci_info() and remove_dev_pci_info(); use pci_sriov_get_totalvfs() instead * use dev_warn() instead of pr_warn() when possible * check to be sure IOV BAR is still in range after shifting, change pnv_pci_vf_resource_shift() from void to int * improve sriov_e
