Re: [RFC 1/3] mm: iommu: An API to unify IOMMU, CPU and device memory management
On Thu, Jul 01, 2010 at 12:16:18AM -0700, Zach Pfeffer wrote: > Thank you for the corrections. I'm correcting them now. Some responses: > > Randy Dunlap wrote: > >> +struct vcm *vcm_create(size_t start_addr, size_t len); > > > > Seems odd to use size_t for start_addr. > > I used size_t because I wanted to allow the start_addr the same range > as len. Is there a better type to use? I see 'unsigned long' used > throughout the mm code. Perhaps that's better for both the start_addr > and len. > phys_addr_t or resource_size_t. -- To unsubscribe from this list: send the line "unsubscribe linux-omap" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC 1/3] mm: iommu: An API to unify IOMMU, CPU and device memory management
Thank you for the corrections. I'm correcting them now. Some responses: Randy Dunlap wrote: >> +struct vcm *vcm_create(size_t start_addr, size_t len); > > Seems odd to use size_t for start_addr. I used size_t because I wanted to allow the start_addr the same range as len. Is there a better type to use? I see 'unsigned long' used throughout the mm code. Perhaps that's better for both the start_addr and len. >> +A Reservation is created and destroyed with: >> + >> +struct res *vcm_reserve(struct vcm *vcm, size_t len, uint32_t attr); > > s/uint32_t/u32/ ? Sure. >> +Associate and activate all three to their respective devices: >> + >> +avcm_iommu = vcm_assoc(vcm_iommu, dev_iommu, attr0); >> +avcm_onetoone = vcm_assoc(vcm_onetoone, dev_onetoone, attr1); >> +avcm_vmm = vcm_assoc(vcm_vmm, dev_cpu, attr2); > > error handling on vcm_assoc() failures? I'll add the deassociate call to the example. >> +res_iommu = vcm_reserve(vcm_iommu, SZ_2MB + SZ_4K, attr); >> +res_onetoone = vcm_reserve(vcm_onetoone, SZ_2MB + SZ_4K, attr); >> +res_vmm = vcm_reserve(vcm_vmm, SZ_2MB + SZ_4K, attr); > > error handling? I'll add it here too. -- Sent by an employee of the Qualcomm Innovation Center, Inc. The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum. -- To unsubscribe from this list: send the line "unsubscribe linux-omap" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC 1/3] mm: iommu: An API to unify IOMMU, CPU and device memory management
On Tue, 29 Jun 2010 22:55:48 -0700 Zach Pfeffer wrote: > This patch contains the documentation for the API, termed the Virtual > Contiguous Memory Manager. Its use would allow all of the IOMMU to VM, > VM to device and device to IOMMU interoperation code to be refactored > into platform independent code. > > Comments, suggestions and criticisms are welcome and wanted. > > Signed-off-by: Zach Pfeffer > --- > Documentation/vcm.txt | 583 > + > 1 files changed, 583 insertions(+), 0 deletions(-) > create mode 100644 Documentation/vcm.txt > > diff --git a/Documentation/vcm.txt b/Documentation/vcm.txt > new file mode 100644 > index 000..d29c757 > --- /dev/null > +++ b/Documentation/vcm.txt > @@ -0,0 +1,583 @@ > +What is this document about? > + > + > +This document covers how to use the Virtual Contiguous Memory Manager > +(VCMM), how the first implmentation works with a specific low-level > +Input/Output Memory Management Unit (IOMMU) and the way the VCMM is used > +from user-space. It also contains a section that describes why something > +like the VCMM is needed in the kernel. > + > +If anything in this document is wrong please send patches to the is wrong, > +maintainer of this file, listed at the bottom of the document. > + > + > +The Virtual Contiguous Memory Manager > += > + > +The VCMM was built to solve the system-wide memory mapping issues that > +occur when many bus-masters have IOMMUs. > + > +An IOMMU maps device addresses to physical addresses. It also insulates > +the system from spurious or malicious device bus transactions and allows > +fine-grained mapping attribute control. The Linux kernel core does not > +contain a generic API to handle IOMMU mapped memory; device driver writers > +must implement device specific code to interoperate with the Linux kernel > +core. As the number of IOMMUs increases, coordinating the many address > +spaces mapped by all discrete IOMMUs becomes difficult without in-kernel > +support. > + > +The VCMM API enables device independent IOMMU control, virtual memory > +manager (VMM) interoperation and non-IOMMU enabled device interoperation > +by treating devices with or without IOMMUs and all CPUs with or without > +MMUs, their mapping contexts and their mappings using common > +abstractions. Physical hardware is given a generic device type and mapping > +contexts are abstracted into Virtual Contiguous Memory (VCM) > +regions. Users "reserve" memory from VCMs and "back" their reservations > +with physical memory. > + > +Why the VCMM is Needed > +-- > + > +Driver writers who control devices with IOMMUs must contend with device > +control and memory management. Driver writers have a large device driver > +API that they can leverage to control their devices, but they are lacking > +a unified API to help them program mappings into IOMMUs and share those > +mappings with other devices and CPUs in the system. > + > +Sharing is complicated by Linux's CPU centric VMM. The CPU centric model CPU-centric CPU-centric > +generally makes sense because average hardware only contains a MMU for the > +CPU and possibly a graphics MMU. If every device in the system has one or > +more MMUs the CPU centric memory management (MM) programming model breaks ditto > +down. > + > +Abstracting IOMMU device programming into a common API has already begun > +in the Linux kernel. It was built to abstract the difference between AMDs AMD's (or just "AMD") > +and Intels IOMMUs to support x86 virtualization on both platforms. The Intel's (or just "Intel") > +interface is listed in kernel/include/linux/iommu.h. It contains drop "kernel/" > +interfaces for mapping and unmapping as well as domain management. This > +interface has not gained widespread use outside the x86; PA-RISC, Alpha > +and SPARC architectures and ARM and PowerPC platforms all use their own > +mapping modules to control their IOMMUs. The VCMM contains an IOMMU > +programming layer, but since its abstraction supports map management > +independent of device control, the layer is not used directly. This > +higher-level view enables a new kernel service, not just an IOMMU > +interoperation layer. > + > +The General Idea: Map Management using Graphs > +- > + > +Looking at mapping from a system-wide perspective reveals a general graph > +problem. The VCMMs API is built to manage the general mapping graph. Each VMCC's > +node that talks to memory, either through an MMU or directly (physically > +mapped) can be thought of as the device-end of a mapping edge. The other > +edge is the physical memory (or intermediate virtual space) tha
[RFC 1/3] mm: iommu: An API to unify IOMMU, CPU and device memory management
This patch contains the documentation for the API, termed the Virtual Contiguous Memory Manager. Its use would allow all of the IOMMU to VM, VM to device and device to IOMMU interoperation code to be refactored into platform independent code. Comments, suggestions and criticisms are welcome and wanted. Signed-off-by: Zach Pfeffer --- Documentation/vcm.txt | 583 + 1 files changed, 583 insertions(+), 0 deletions(-) create mode 100644 Documentation/vcm.txt diff --git a/Documentation/vcm.txt b/Documentation/vcm.txt new file mode 100644 index 000..d29c757 --- /dev/null +++ b/Documentation/vcm.txt @@ -0,0 +1,583 @@ +What is this document about? + + +This document covers how to use the Virtual Contiguous Memory Manager +(VCMM), how the first implmentation works with a specific low-level +Input/Output Memory Management Unit (IOMMU) and the way the VCMM is used +from user-space. It also contains a section that describes why something +like the VCMM is needed in the kernel. + +If anything in this document is wrong please send patches to the +maintainer of this file, listed at the bottom of the document. + + +The Virtual Contiguous Memory Manager += + +The VCMM was built to solve the system-wide memory mapping issues that +occur when many bus-masters have IOMMUs. + +An IOMMU maps device addresses to physical addresses. It also insulates +the system from spurious or malicious device bus transactions and allows +fine-grained mapping attribute control. The Linux kernel core does not +contain a generic API to handle IOMMU mapped memory; device driver writers +must implement device specific code to interoperate with the Linux kernel +core. As the number of IOMMUs increases, coordinating the many address +spaces mapped by all discrete IOMMUs becomes difficult without in-kernel +support. + +The VCMM API enables device independent IOMMU control, virtual memory +manager (VMM) interoperation and non-IOMMU enabled device interoperation +by treating devices with or without IOMMUs and all CPUs with or without +MMUs, their mapping contexts and their mappings using common +abstractions. Physical hardware is given a generic device type and mapping +contexts are abstracted into Virtual Contiguous Memory (VCM) +regions. Users "reserve" memory from VCMs and "back" their reservations +with physical memory. + +Why the VCMM is Needed +-- + +Driver writers who control devices with IOMMUs must contend with device +control and memory management. Driver writers have a large device driver +API that they can leverage to control their devices, but they are lacking +a unified API to help them program mappings into IOMMUs and share those +mappings with other devices and CPUs in the system. + +Sharing is complicated by Linux's CPU centric VMM. The CPU centric model +generally makes sense because average hardware only contains a MMU for the +CPU and possibly a graphics MMU. If every device in the system has one or +more MMUs the CPU centric memory management (MM) programming model breaks +down. + +Abstracting IOMMU device programming into a common API has already begun +in the Linux kernel. It was built to abstract the difference between AMDs +and Intels IOMMUs to support x86 virtualization on both platforms. The +interface is listed in kernel/include/linux/iommu.h. It contains +interfaces for mapping and unmapping as well as domain management. This +interface has not gained widespread use outside the x86; PA-RISC, Alpha +and SPARC architectures and ARM and PowerPC platforms all use their own +mapping modules to control their IOMMUs. The VCMM contains an IOMMU +programming layer, but since its abstraction supports map management +independent of device control, the layer is not used directly. This +higher-level view enables a new kernel service, not just an IOMMU +interoperation layer. + +The General Idea: Map Management using Graphs +- + +Looking at mapping from a system-wide perspective reveals a general graph +problem. The VCMMs API is built to manage the general mapping graph. Each +node that talks to memory, either through an MMU or directly (physically +mapped) can be thought of as the device-end of a mapping edge. The other +edge is the physical memory (or intermediate virtual space) that is +mapped. + +In the direct mapped case the device is assigned a one-to-one MMU. This +scheme allows direct mapped devices to participate in general graph +management. + +The CPU nodes can also be brought under the same mapping abstraction with +the use of a light overlay on the existing VMM. This light overlay allows +VMM managed mappings to interoperate with the common API. The light +overlay enables this without substantial modifications to the existing +VMM. + +In addition to CPU nodes that are running Linux (and the VMM), remote CPU +nodes that may be running other ope
