Re: RFC: vfio API changes needed for powerpc (v3)
On Tue, Apr 09, 2013 at 01:22:15AM +, Yoder Stuart-B08248 wrote: What happens if a normal unmap call is done on the MSI iova? Do we need a separate unmap? I was thinking a normal unmap on an MSI windows would be an error...but I'm not set on that. I put the msi unmap there to make things symmetric, a normal unmap would work as well...and then we could drop the msi unmap. Hmm, this API semantic isn't very clean. When you explicitly map the MSI banks a clean API would also allow to unmap them. But that is not possible in your design because the kernel is responsible for mapping MSIs and you can't unmap a MSI bank that is in use by the kernel. So since the kernel owns the MSI setup anyways it should also take care of mapping the MSI banks. What is the reason to not let the kernel allocate the MSI banks top-down from the end of the DMA window space? Just let userspace know (or even set if needed) in advance how many of the windows it configures the kernel will take for mapping MSI banks and you are fine, no? Joerg -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: RFC: vfio API changes needed for powerpc (v3)
-Original Message- From: Joerg Roedel [mailto:j...@8bytes.org] Sent: Thursday, April 11, 2013 7:57 AM To: Yoder Stuart-B08248 Cc: Wood Scott-B07421; kvm@vger.kernel.org; qemu-de...@nongnu.org; io...@lists.linux-foundation.org; ag...@suse.de; Bhushan Bharat-R65777 Subject: Re: RFC: vfio API changes needed for powerpc (v3) On Tue, Apr 09, 2013 at 01:22:15AM +, Yoder Stuart-B08248 wrote: What happens if a normal unmap call is done on the MSI iova? Do we need a separate unmap? I was thinking a normal unmap on an MSI windows would be an error...but I'm not set on that. I put the msi unmap there to make things symmetric, a normal unmap would work as well...and then we could drop the msi unmap. Hmm, this API semantic isn't very clean. When you explicitly map the MSI banks a clean API would also allow to unmap them. But that is not possible in your design because the kernel is responsible for mapping MSIs and you can't unmap a MSI bank that is in use by the kernel. The mapping that the vfio API creates is specific only to the assigned device. So it can be unmapped without affecting any other device... there is nothing else in the kernel making the mapping in use. Another device in use by the kernel using the same MSI bank would have its own independent mapping. So, these mappings are not global but are device specific...just like normal DMA memory mappings. So since the kernel owns the MSI setup anyways it should also take care of mapping the MSI banks. What is the reason to not let the kernel allocate the MSI banks top-down from the end of the DMA window space? Just let userspace know (or even set if needed) in advance how many of the windows it configures the kernel will take for mapping MSI banks and you are fine, no? As designed the API lets user space determine the number of windows needed for MSIs, so they can be set. The only difference between what we've proposed and what you described, I think, is that the proposal allows user space to _which_ windows are used for which MSI banks. Stuart -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: RFC: vfio API changes needed for powerpc (v3)
On 04/11/2013 07:56:59 AM, Joerg Roedel wrote: On Tue, Apr 09, 2013 at 01:22:15AM +, Yoder Stuart-B08248 wrote: What happens if a normal unmap call is done on the MSI iova? Do we need a separate unmap? I was thinking a normal unmap on an MSI windows would be an error...but I'm not set on that. I put the msi unmap there to make things symmetric, a normal unmap would work as well...and then we could drop the msi unmap. Hmm, this API semantic isn't very clean. When you explicitly map the MSI banks a clean API would also allow to unmap them. But that is not possible in your design because the kernel is responsible for mapping MSIs and you can't unmap a MSI bank that is in use by the kernel. Why is it not possible to unmap them? Once they've been mapped, they're just like any other IOMMU mapping. If the user breaks MSI for their own devices by unmapping the MSI page, that's their problem. So since the kernel owns the MSI setup anyways it should also take care of mapping the MSI banks. What is the reason to not let the kernel allocate the MSI banks top-down from the end of the DMA window space? It's less flexible, and possibly more complicated. -Scott -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: RFC: vfio API changes needed for powerpc (v3)
So now the sequence would be something like:
1)VFIO_GROUP_SET_CONTAINER // add groups to the container
2)VFIO_SET_IOMMU(VFIO_FSL_PAMU)// set iommu model
3)count = VFIO_IOMMU_GET_MSI_BANK_COUNT// returns max # of MSI banks
4)VFIO_IOMMU_SET_ATTR(ATTR_GEOMETRY) // set overall aperture
5)VFIO_IOMMU_SET_ATTR(ATTR_WINDOWS) // set # of windows, including MSI
banks
6) For (int I = 0; I count; i++)
VFIO_IOMMU_PAMU_MAP_MSI_BANK() // map the MSI banks, do not
enable aperture here.
7) Memory Listener will call- VFIO_IOMMU_MAP_DMA// map the guest's
memory
--- kernel enables aperture here on first VFIO_IOMMU_MAP_DMA
8)VFIO_DEVICE_SET_IRQS
--- VFIO in kernel makes
pci_enable_msix()/pci_enable_msi_block() calls, this sets actual MSI addr/data
in physical device.
--- As the address set by above APIs is not what we want so
- is using MSIX, VFIO will update address in the MSI-X
table
- if using MSI, update MSI address in PCI configuration
space.
Thanks
-Bharat
-Original Message-
From: Yoder Stuart-B08248
Sent: Friday, April 05, 2013 3:40 AM
To: Alex Williamson
Cc: Wood Scott-B07421; ag...@suse.de; Bhushan Bharat-R65777; Sethi
Varun-B16395;
kvm@vger.kernel.org; qemu-de...@nongnu.org; io...@lists.linux-foundation.org
Subject: RFC: vfio API changes needed for powerpc (v3)
-v3 updates
-made vfio_pamu_attr a union, added flags
-s/VFIO_PAMU_/VFIO_IOMMU_PAMU_/ for the ioctls to make it more
clear which fd is being operated on
-added flags to vfio_pamu_msi_bank_map/umap
-VFIO_PAMU_GET_MSI_BANK_COUNT now just returns a __u32
not a struct
-fixed some typos
The Freescale PAMU is an aperture-based IOMMU with the following
characteristics. Each device has an entry in a table in memory
describing the iova-phys mapping. The mapping has:
-an overall aperture that is power of 2 sized, and has a start iova that
is naturally aligned
-has 1 or more windows within the aperture
-number of windows must be power of 2, max is 256
-size of each window is determined by aperture size / # of windows
-iova of each window is determined by aperture start iova / # of windows
-the mapped region in each window can be different than
the window size...mapping must power of 2
-physical address of the mapping must be naturally aligned
with the mapping size
These ioctls operate on the VFIO file descriptor (/dev/vfio/vfio).
/*
* VFIO_IOMMU_PAMU_GET_ATTR
*
* Gets the iommu attributes for the current vfio container. This
* ioctl is applicable to an iommu type of VFIO_PAMU only.
* Caller sets argsz and attribute. The ioctl fills in
* the provided struct vfio_pamu_attr based on the attribute
* value that was set.
* Return: 0 on success, -errno on failure
*/
struct vfio_pamu_attr {
__u32 argsz;
__u32 flags;/* no flags currently */
__u32 attribute;
union {
/* VFIO_ATTR_GEOMETRY */
struct {
__u64 aperture_start; /* first addr that can be mapped
*/
__u64 aperture_end; /* last addr that can be mapped
*/
} attr;
/* VFIO_ATTR_WINDOWS */
__u32 windows; /* number of windows in the aperture */
/* initially this will be the max number
* of windows that can be set
*/
/* VFIO_ATTR_PAMU_STASH */
struct {
__u32 cpu; /* CPU number for stashing */
__u32 cache; /* cache ID for stashing */
} stash;
}
};
#define VFIO_IOMMU_PAMU_GET_ATTR _IO(VFIO_TYPE, VFIO_BASE + x,
struct vfio_pamu_attr)
/*
* VFIO_IOMMU_PAMU_SET_ATTR
*
* Sets the iommu attributes for the current vfio container. This
* ioctl is applicable to an iommu type of VFIO_PAMU only.
* Caller sets struct vfio_pamu attr, including argsz and attribute and
* setting any fields that are valid for the attribute.
* Return: 0 on success, -errno on failure
*/
#define VFIO_IOMMU_PAMU_SET_ATTR _IO(VFIO_TYPE, VFIO_BASE + x,
struct vfio_pamu_attr)
/*
* VFIO_IOMMU_PAMU_GET_MSI_BANK_COUNT
*
* Returns the number of MSI banks for this platform. This tells user space
* how many aperture windows should be reserved for MSI banks when setting
* the PAMU geometry and window count.
* Return: __u32 bank count on success, -errno on failure
*/
#define VFIO_IOMMU_PAMU_GET_MSI_BANK_COUNT _IO(VFIO_TYPE, VFIO_BASE + x,
__u32)
/*
* VFIO_IOMMU_PAMU_MAP_MSI_BANK
*
RE: RFC: vfio API changes needed for powerpc (v3)
-Original Message-
From: Wood Scott-B07421
Sent: Friday, April 05, 2013 5:17 PM
To: Yoder Stuart-B08248
Cc: Alex Williamson; Wood Scott-B07421; ag...@suse.de; Bhushan Bharat-R65777;
Sethi Varun-B16395;
kvm@vger.kernel.org; qemu-de...@nongnu.org; io...@lists.linux-foundation.org
Subject: Re: RFC: vfio API changes needed for powerpc (v3)
On 04/04/2013 05:10:27 PM, Yoder Stuart-B08248 wrote:
/*
* VFIO_IOMMU_PAMU_UNMAP_MSI_BANK
*
* Unmaps the MSI bank at the specified iova.
* Caller provides struct vfio_pamu_msi_bank_unmap with all fields
set.
* Operates on VFIO file descriptor (/dev/vfio/vfio).
* Return: 0 on success, -errno on failure
*/
struct vfio_pamu_msi_bank_unmap {
__u32 argsz;
__u32 flags; /* no flags currently */
__u64 iova; /* the iova to be unmapped to */
};
#define VFIO_IOMMU_PAMU_UNMAP_MSI_BANK _IO(VFIO_TYPE, VFIO_BASE + x,
struct vfio_pamu_msi_bank_unmap )
What happens if a normal unmap call is done on the MSI iova? Do we
need a separate unmap?
I was thinking a normal unmap on an MSI windows would be an error...but
I'm not set on that. I put the msi unmap there to make things symmetric,
a normal unmap would work as well...and then we could drop the msi unmap.
Stuart
--
To unsubscribe from this list: send the line unsubscribe kvm in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: RFC: vfio API changes needed for powerpc (v3)
On 04/04/2013 05:10:27 PM, Yoder Stuart-B08248 wrote:
/*
* VFIO_IOMMU_PAMU_UNMAP_MSI_BANK
*
* Unmaps the MSI bank at the specified iova.
* Caller provides struct vfio_pamu_msi_bank_unmap with all fields
set.
* Operates on VFIO file descriptor (/dev/vfio/vfio).
* Return: 0 on success, -errno on failure
*/
struct vfio_pamu_msi_bank_unmap {
__u32 argsz;
__u32 flags; /* no flags currently */
__u64 iova; /* the iova to be unmapped to */
};
#define VFIO_IOMMU_PAMU_UNMAP_MSI_BANK _IO(VFIO_TYPE, VFIO_BASE + x,
struct vfio_pamu_msi_bank_unmap )
What happens if a normal unmap call is done on the MSI iova? Do we
need a separate unmap?
-Scott
--
To unsubscribe from this list: send the line unsubscribe kvm in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
