Re: [Qemu-devel] [PATCH v5 00/18] VT-d: vfio enablement and misc enhances
On Wed, Jan 25, 2017 at 04:26:59PM +0200, Michael S. Tsirkin wrote: > On Wed, Jan 25, 2017 at 12:48:37PM +0800, Peter Xu wrote: > > On Tue, Jan 24, 2017 at 04:48:49PM +0200, Michael S. Tsirkin wrote: > > > On Tue, Jan 24, 2017 at 07:49:10PM +0800, Peter Xu wrote: > > > > On Tue, Jan 24, 2017 at 06:25:53PM +0800, Peter Xu wrote: > > > > > This is v5 of vt-d vfio enablement series. > > > > > > > > > > Most of the changes in v5 is not functionally, but related to > > > > > comments, error_report()s, debugging, squashing patches, etc. (which > > > > > are confirmed changes in v4), besides a tiny tweak when unmapping a > > > > > whole address space (please see below changelog). There are still > > > > > discussions in v4 thread, we can just continue there (or here), and > > > > > from this version I'll remove RFC tag. > > > > > > > > > > I didn't rebase to master since current master failed to build on my > > > > > laptop (with a "vl.c/hax..." error), however this series should be > > > > > okay to be applied cleanly upon it. > > > > > > > > Sorry I forgot to append the todo list (please help adding in in case > > > > I missed anything): > > > > > > > > - don't need to notify IOTLB (psi/gsi/global) invalidations to devices > > > > that with ATS enabled > > > > - investigate when guest map page while mask contains existing mapped > > > > pages (e.g. map 12k-16k first, then map 0-12k) > > > > - coalesce unmap during page walk (currently, we send it once per > > > > page) > > > > - when do PSI for unmap, whether we can send one notify directly > > > > instead of walking over the page table? > > > > > > > > Above does not contain those that are still during discussion. And, > > > > some of the entries still need tests to further prove it's > > > > feasibility. > > > > > > > > Thanks, > > > > > > > > -- peterx > > > > > > While valid I don't think above need to block merging. > > > > Sorry for the delay of this series. > > > > Looks like we still need to wait until we got explicit acks for vfio > > patch 2-3 from Alex/Paolo/David since that'll be required by the > > following series. (I'll address all Jason's new comments as well in > > next post) > > > > Do you like to merge the iommu cleanups first? Or you can also wait > > for above to be confirmed, and I'll repost for a final version then > > (if we don't have further comments). > > > > Thanks, > > > > -- peterx > > I can do that. If you want me to, pls post just the patches > that are ready separately. Thanks for the offer! Since I see that currently the only uncertainty of this series would be the impact that patch 3 might bring to Power, I'll prefer we wait for several more days until David's back (I suppose it's next Monday or so), to see whether I can send a complete final version. Thanks, -- peterx
Re: [Qemu-devel] [PATCH v5 00/18] VT-d: vfio enablement and misc enhances
On Wed, Jan 25, 2017 at 12:48:37PM +0800, Peter Xu wrote: > On Tue, Jan 24, 2017 at 04:48:49PM +0200, Michael S. Tsirkin wrote: > > On Tue, Jan 24, 2017 at 07:49:10PM +0800, Peter Xu wrote: > > > On Tue, Jan 24, 2017 at 06:25:53PM +0800, Peter Xu wrote: > > > > This is v5 of vt-d vfio enablement series. > > > > > > > > Most of the changes in v5 is not functionally, but related to > > > > comments, error_report()s, debugging, squashing patches, etc. (which > > > > are confirmed changes in v4), besides a tiny tweak when unmapping a > > > > whole address space (please see below changelog). There are still > > > > discussions in v4 thread, we can just continue there (or here), and > > > > from this version I'll remove RFC tag. > > > > > > > > I didn't rebase to master since current master failed to build on my > > > > laptop (with a "vl.c/hax..." error), however this series should be > > > > okay to be applied cleanly upon it. > > > > > > Sorry I forgot to append the todo list (please help adding in in case > > > I missed anything): > > > > > > - don't need to notify IOTLB (psi/gsi/global) invalidations to devices > > > that with ATS enabled > > > - investigate when guest map page while mask contains existing mapped > > > pages (e.g. map 12k-16k first, then map 0-12k) > > > - coalesce unmap during page walk (currently, we send it once per > > > page) > > > - when do PSI for unmap, whether we can send one notify directly > > > instead of walking over the page table? > > > > > > Above does not contain those that are still during discussion. And, > > > some of the entries still need tests to further prove it's > > > feasibility. > > > > > > Thanks, > > > > > > -- peterx > > > > While valid I don't think above need to block merging. > > Sorry for the delay of this series. > > Looks like we still need to wait until we got explicit acks for vfio > patch 2-3 from Alex/Paolo/David since that'll be required by the > following series. (I'll address all Jason's new comments as well in > next post) > > Do you like to merge the iommu cleanups first? Or you can also wait > for above to be confirmed, and I'll repost for a final version then > (if we don't have further comments). > > Thanks, > > -- peterx I can do that. If you want me to, pls post just the patches that are ready separately. -- MST
Re: [Qemu-devel] [PATCH v5 00/18] VT-d: vfio enablement and misc enhances
On Tue, Jan 24, 2017 at 04:48:49PM +0200, Michael S. Tsirkin wrote: > On Tue, Jan 24, 2017 at 07:49:10PM +0800, Peter Xu wrote: > > On Tue, Jan 24, 2017 at 06:25:53PM +0800, Peter Xu wrote: > > > This is v5 of vt-d vfio enablement series. > > > > > > Most of the changes in v5 is not functionally, but related to > > > comments, error_report()s, debugging, squashing patches, etc. (which > > > are confirmed changes in v4), besides a tiny tweak when unmapping a > > > whole address space (please see below changelog). There are still > > > discussions in v4 thread, we can just continue there (or here), and > > > from this version I'll remove RFC tag. > > > > > > I didn't rebase to master since current master failed to build on my > > > laptop (with a "vl.c/hax..." error), however this series should be > > > okay to be applied cleanly upon it. > > > > Sorry I forgot to append the todo list (please help adding in in case > > I missed anything): > > > > - don't need to notify IOTLB (psi/gsi/global) invalidations to devices > > that with ATS enabled > > - investigate when guest map page while mask contains existing mapped > > pages (e.g. map 12k-16k first, then map 0-12k) > > - coalesce unmap during page walk (currently, we send it once per > > page) > > - when do PSI for unmap, whether we can send one notify directly > > instead of walking over the page table? > > > > Above does not contain those that are still during discussion. And, > > some of the entries still need tests to further prove it's > > feasibility. > > > > Thanks, > > > > -- peterx > > While valid I don't think above need to block merging. Sorry for the delay of this series. Looks like we still need to wait until we got explicit acks for vfio patch 2-3 from Alex/Paolo/David since that'll be required by the following series. (I'll address all Jason's new comments as well in next post) Do you like to merge the iommu cleanups first? Or you can also wait for above to be confirmed, and I'll repost for a final version then (if we don't have further comments). Thanks, -- peterx
Re: [Qemu-devel] [PATCH v5 00/18] VT-d: vfio enablement and misc enhances
On Tue, Jan 24, 2017 at 07:49:10PM +0800, Peter Xu wrote: > On Tue, Jan 24, 2017 at 06:25:53PM +0800, Peter Xu wrote: > > This is v5 of vt-d vfio enablement series. > > > > Most of the changes in v5 is not functionally, but related to > > comments, error_report()s, debugging, squashing patches, etc. (which > > are confirmed changes in v4), besides a tiny tweak when unmapping a > > whole address space (please see below changelog). There are still > > discussions in v4 thread, we can just continue there (or here), and > > from this version I'll remove RFC tag. > > > > I didn't rebase to master since current master failed to build on my > > laptop (with a "vl.c/hax..." error), however this series should be > > okay to be applied cleanly upon it. > > Sorry I forgot to append the todo list (please help adding in in case > I missed anything): > > - don't need to notify IOTLB (psi/gsi/global) invalidations to devices > that with ATS enabled > - investigate when guest map page while mask contains existing mapped > pages (e.g. map 12k-16k first, then map 0-12k) > - coalesce unmap during page walk (currently, we send it once per > page) > - when do PSI for unmap, whether we can send one notify directly > instead of walking over the page table? > > Above does not contain those that are still during discussion. And, > some of the entries still need tests to further prove it's > feasibility. > > Thanks, > > -- peterx While valid I don't think above need to block merging. -- MST
Re: [Qemu-devel] [PATCH v5 00/18] VT-d: vfio enablement and misc enhances
On Tue, Jan 24, 2017 at 06:25:53PM +0800, Peter Xu wrote: > This is v5 of vt-d vfio enablement series. > > Most of the changes in v5 is not functionally, but related to > comments, error_report()s, debugging, squashing patches, etc. (which > are confirmed changes in v4), besides a tiny tweak when unmapping a > whole address space (please see below changelog). There are still > discussions in v4 thread, we can just continue there (or here), and > from this version I'll remove RFC tag. > > I didn't rebase to master since current master failed to build on my > laptop (with a "vl.c/hax..." error), however this series should be > okay to be applied cleanly upon it. Sorry I forgot to append the todo list (please help adding in in case I missed anything): - don't need to notify IOTLB (psi/gsi/global) invalidations to devices that with ATS enabled - investigate when guest map page while mask contains existing mapped pages (e.g. map 12k-16k first, then map 0-12k) - coalesce unmap during page walk (currently, we send it once per page) - when do PSI for unmap, whether we can send one notify directly instead of walking over the page table? Above does not contain those that are still during discussion. And, some of the entries still need tests to further prove it's feasibility. Thanks, -- peterx
[Qemu-devel] [PATCH v5 00/18] VT-d: vfio enablement and misc enhances
This is v5 of vt-d vfio enablement series.
Most of the changes in v5 is not functionally, but related to
comments, error_report()s, debugging, squashing patches, etc. (which
are confirmed changes in v4), besides a tiny tweak when unmapping a
whole address space (please see below changelog). There are still
discussions in v4 thread, we can just continue there (or here), and
from this version I'll remove RFC tag.
I didn't rebase to master since current master failed to build on my
laptop (with a "vl.c/hax..." error), however this series should be
okay to be applied cleanly upon it.
v5:
- fix patch 4 subject too long, and error spelling [Eric]
- add ack-by for alex in patch 1 [Alex]
- squashing patch 19/20 into patch 18 [Jason]
- fix comments in vtd_page_walk() [Jason]
- remove all error_report() [Jason]
- add comment for patch 18, mention about that enabled vhost without
ATS as well [Jason]
- remove skipped debug thing during page walk [Jason]
- remove duplicated page walk trace [Jason]
- some tunings in vtd_address_space_unmap(), to provide correct iova
and addr_mask. For this, I tuned this patch as well a bit:
"memory: add section range info for IOMMU notifier"
to loosen the range check
v4:
- convert all error_report()s into traces (in the two patches that did
that)
- rebased to Jason's DMAR series (master + one more patch:
"[PATCH V4 net-next] vhost_net: device IOTLB support")
- let vhost use the new api iommu_notifier_init() so it won't break
vhost dmar [Jason]
- touch commit message of the patch:
"intel_iommu: provide its own replay() callback"
old replay is not a dead loop, but it will just consume lots of time
[Jason]
- add comment for patch:
"intel_iommu: do replay when context invalidate"
telling why replay won't be a problem even without CM=1 [Jason]
- remove a useless comment line [Jason]
- remove dmar_enabled parameter for vtd_switch_address_space() and
vtd_switch_address_space_all() [Mst, Jason]
- merged the vfio patches in, to support unmap of big ranges at the
beginning ("[PATCH RFC 0/3] vfio: allow to notify unmap for very big
region")
- using caching_mode instead of cache_mode_enabled, and "caching-mode"
instead of "cache-mode" [Kevin]
- when receive context entry invalidation, we unmap the entire region
first, then replay [Alex]
- fix commit message for patch:
"intel_iommu: simplify irq region translation" [Kevin]
- handle domain/global invalidation, and notify where proper [Jason,
Kevin]
v3:
- fix style error reported by patchew
- fix comment in domain switch patch: use "IOMMU address space" rather
than "IOMMU region" [Kevin]
- add ack-by for Paolo in patch:
"memory: add section range info for IOMMU notifier"
(this is seperately collected besides this thread)
- remove 3 patches which are merged already (from Jason)
- rebase to master b6c0897
v2:
- change comment for "end" parameter in vtd_page_walk() [Tianyu]
- change comment for "a iova" to "an iova" [Yi]
- fix fault printed val for GPA address in vtd_page_walk_level (debug
only)
- rebased to master (rather than Aviv's v6 series) and merged Aviv's
series v6: picked patch 1 (as patch 1 in this series), dropped patch
2, re-wrote patch 3 (as patch 17 of this series).
- picked up two more bugfix patches from Jason's DMAR series
- picked up the following patch as well:
"[PATCH v3] intel_iommu: allow dynamic switch of IOMMU region"
This RFC series is a re-work for Aviv B.D.'s vfio enablement series
with vt-d:
https://lists.gnu.org/archive/html/qemu-devel/2016-11/msg01452.html
Aviv has done a great job there, and what we still lack there are
mostly the following:
(1) VFIO got duplicated IOTLB notifications due to splitted VT-d IOMMU
memory region.
(2) VT-d still haven't provide a correct replay() mechanism (e.g.,
when IOMMU domain switches, things will broke).
This series should have solved the above two issues.
Online repo:
https://github.com/xzpeter/qemu/tree/vtd-vfio-enablement-v5
I would be glad to hear about any review comments for above patches.
=
Test Done
=
Build test passed for x86_64/arm/ppc64.
Simply tested with x86_64, assigning two PCI devices to a single VM,
boot the VM using:
bin=x86_64-softmmu/qemu-system-x86_64
$bin -M q35,accel=kvm,kernel-irqchip=split -m 1G \
-device intel-iommu,intremap=on,eim=off,caching-mode=on \
-netdev user,id=net0,hostfwd=tcp::-:22 \
-device virtio-net-pci,netdev=net0 \
-device vfio-pci,host=03:00.0 \
-device vfio-pci,host=02:00.0 \
-trace events=".trace.vfio" \
/var/lib/libvirt/images/vm1.qcow2
pxdev:bin [vtd-vfio-enablement]# cat .trace.vfio
vtd_page_walk*
vtd_replay*
vtd_inv_desc*
Then, in the guest, run the following tool:
https://github.com/xzpeter/clibs/blob/master/gpl/userspace/vfio-bind-group/vfio-bind-group.c
With parameter:
./vfio-bind-group 00:03.0 00:04.0
Check host side trace log, I can see pages are replayed and mapped in
00:04.0 device address
