On 2016年11月11日 01:32, Michael S. Tsirkin wrote:
On Fri, Nov 04, 2016 at 02:48:20PM +0800, Jason Wang wrote:
On 2016年11月04日 03:49, Michael S. Tsirkin wrote:
On Thu, Nov 03, 2016 at 05:27:19PM +0800, Jason Wang wrote:
This patches enable the Address Translation Service support for virtio
pci devices. This is needed for a guest visible Device IOTLB
implementation and will be required by vhost device IOTLB API
implementation for intel IOMMU.
Cc: Michael S. Tsirkin<m...@redhat.com>
Signed-off-by: Jason Wang<jasow...@redhat.com>
I'd like to understand why do you think this is strictly required.
Won't setting CM bit in the IOMMU do the trick.
ATS was chosen for performance. Since there're many problems for CM:
- CM was slow (10%-20% slower on real hardware for things like netperf)
because of each transition between non-present and present mapping needs an
explicit invalidation. It may slow down the whole VM.
- Without ATS/Device IOTLB, IOMMU becomes a bottleneck because of contending
of IOTLB entries. (What we can do in this case is in fact userspace IOTLB
snooping, this could be done even without CM).
It was natural to think of ATS when designing interface between IOMMU and
device/remote IOTLBs. Do you see any drawbacks on ATS here?
Thanks
In fact at this point I'm confused. Any mapping needs to be programmed
in the IOMMU. We need to implement this correctly.
Once we do why do we need ATS?
I think what you need is map/unmap notifiers that Aviv is working on.
No?
Let me clarify, device IOTLB API can work without ATS or CM. So there're
three ways to do:
1) without ATS or CM support, the function could be implemented through:
1.1: asking for qemu help if there's an IOTLB miss in vhost
1.2: snooping the userspace IOTLB invalidation (present to non-present
mapping) and update device IOTLB
2) with CM enabled, the only thing we can add is snooping the
non-present to present mapping and update the device IOTLB. This is not
a requirement since we still can get this through asking qemu's(1.2) help.
3) with ATS enabled, guest knows the existence of device IOTLB, and
device IOTLB entires needs to be flushed explicitly by guest. In this
case there's no need to snoop the ordinary IOTLB invalidation in 1.2. We
just need to snoop the device IOTLB specific invalidation request from
guest.
All the above 3 methods work very well, but let's have a look at
performance impact:
- Method 1 (without CM or ATS), the performance is not the best since
guest does not know about the existence of remote IOTLB, this means the
flush of device IOTLB entry could not be done on demand. One example is
some IOMMU driver (e.g intel) tends to optimize the IOTLB invalidations
by issuing a global invalidation periodically. We need to flush the
device IOTLB too in this case. Thus we can notice some jitter (because
of IOTLB miss).
- Method 2 (with CM but without ATS) seems to be the worst case. It has
not only all problems above a but also a new one: each transition needs
to notify the device explicitly. Even if dpdk use static mappings, all
other devices in the VM use dynamic ones which slows down the whole the
system. According to the test, CM is about 10%-20% slower in real hardware.
- Method 3 (ATS) can give the best performance, all the problems have
gone since guest can flush the device IOTLB entry on demand. It was
defined by spec and was designed to solve the issues just like what we
meet here, and was supported by modern IOMMUs.
And what's even better, implementing ATS turns out less than 100 lines
of codes. And it was much more easier to be enabled on other IOMMU (AMD
IOMMU only needs 20 lines of codes). All other ways (I started and have
codes for method 1 for intel IOMMU) need lots of work specific to each
kind of IOMMU.
Consider so much advantages by just adding so small lines of codes. I
don't see why we don't need ATS (for the IOOMUs that supports it).
Thanks
Also, could you remind me pls - can guests just disable ATS?
What happens then?
