On Thu, 2016-12-29 at 17:14 -0800, Stephen Hemminger wrote: > If kernel broke pinning of hugepages, then it is an upstream kernel bug.
The kernel, under a myriad of circumstances, will change the mapping of virtual to physical addresses for hugepages. This behavior began somewhere around kernel 3.16 and with each release more cases where the mapping can change are introduced. DPDK should not be relying on that mapping staying static, and instead should be using vfio to explicitly pin the pages. I've consulted the relevant kernel developers who write the code in this area and they are universally in agreement that this is not a kernel bug and the mappings will get less static over time. On Mon, 2017-01-02 at 11:47 -0800, Stephen Hemminger wrote: > On Mon, 02 Jan 2017 15:32:08 +0100 > Thomas Monjalon <thomas.monja...@6wind.com> wrote: > > > 2016-12-29 17:14, Stephen Hemminger: > > > On Thu, 29 Dec 2016 20:41:21 +0000 > > > "Walker, Benjamin" <benjamin.wal...@intel.com> wrote: > > > > My second question is whether the user should be allowed to > > > > mix uio and vfio usage simultaneously. For vfio, the > > > > physical addresses are really DMA addresses and are best > > > > when arbitrarily chosen to appear sequential relative to > > > > their virtual addresses. For uio, they are physical > > > > addresses and are not chosen at all. It seems that these two > > > > things are in conflict and that it will be difficult, ugly, > > > > and maybe impossible to resolve the simultaneous use of > > > > both. > > > > > > Unless application is running as privileged user (ie root), UIO > > > is not going to work. Therefore don't worry about mixed environment. > > > > Yes, mixing UIO and VFIO is possible only as root. > > However, what is the benefit of mixing them? > > One possible case where this could be used, Hyper-V/Azure and SR-IOV. > The VF interface will show up on an isolated PCI bus and the virtual NIC > is on VMBUS. It is possible to use VFIO on the PCI to get MSI-X per queue > interrupts, but there is no support for VFIO on VMBUS. I sent out a patch a little while ago that makes DPDK work when running as an unprivileged user with an IOMMU. I allow mixing of uio/vfio when root (I choose the DMA address to be the physical address), but only vfio when unprivileged (I choose the DMA addresses to start at 0). Unfortunately, there are a few more wrinkles for systems that do not have an IOMMU. These systems still need to explicitly pin memory, but they need to use physical addresses instead of DMA addresses. There are two concerns with this: 1) Physical addresses cannot be exposed to unprivileged users due to security concerns (the fallout of rowhammer). Therefore, systems without an IOMMU can only support privileged users. I think this is probably fine. 2) The IOCTL from vfio to pin the memory is tied to specifying the DMA address and programming the IOMMU. This is unfortunate - systems without an IOMMU still want to do the pinning, but they need to be given the physical address instead of specifying a DMA address. 3) Not all device types, particularly in virtualization environments, support vfio today. These devices have no way to explicitly pin memory. I think this is going to take a kernel patch or two to resolve, unless someone has a good idea.
