Re: [RFC 3/3] mm: iommu: The Virtual Contiguous Memory Manager
On Wed, Jul 21, 2010 at 08:50:26PM -0700, Zach Pfeffer wrote: On Wed, Jul 14, 2010 at 10:59:43AM +0900, FUJITA Tomonori wrote: On Tue, 13 Jul 2010 10:02:23 +0100 Zach Pfeffer said this new VCM infrastructure can be useful for video4linux. However, I don't think we need 3,000-lines another abstraction layer to solve video4linux's issue nicely. Its only 3000 lines because I haven't converted the code to use function pointers. I don't understand - you've made this claim a couple of times. I can't see how converting the code to use function pointers (presumably to eliminate those switch statements) would reduce the number of lines of code. Please explain (or show via new patches) how does converting this to function pointers significantly reduce the number of lines of code. We might then be able to put just _one_ of these issues to bed. Getting back to the point. There is no API that can handle large buffer allocation and sharing with low-level attribute control for virtual address spaces outside the CPU. I think we've dealt with the attribute issue to death now. Shall we repeat it again? The DMA API et al. take a CPU centric view of virtual space management, sharing has to be explicitly written and external virtual space management is left up to device driver writers. I think I've also shown that not to be the case with example code. The code behind the DMA API can be changed on a per-device basis (currently on ARM we haven't supported that because no one's asked for it yet) so that it can support multiple IOMMUs even of multiple different types. -- To unsubscribe from this list: send the line unsubscribe linux-omap in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC 3/3] mm: iommu: The Virtual Contiguous Memory Manager
On Thu, Jul 22, 2010 at 08:51:51AM +0100, Russell King - ARM Linux wrote: On Wed, Jul 21, 2010 at 08:50:26PM -0700, Zach Pfeffer wrote: On Wed, Jul 14, 2010 at 10:59:43AM +0900, FUJITA Tomonori wrote: On Tue, 13 Jul 2010 10:02:23 +0100 Zach Pfeffer said this new VCM infrastructure can be useful for video4linux. However, I don't think we need 3,000-lines another abstraction layer to solve video4linux's issue nicely. Its only 3000 lines because I haven't converted the code to use function pointers. I don't understand - you've made this claim a couple of times. I can't see how converting the code to use function pointers (presumably to eliminate those switch statements) would reduce the number of lines of code. Please explain (or show via new patches) how does converting this to function pointers significantly reduce the number of lines of code. We might then be able to put just _one_ of these issues to bed. Aye. Its getting worked on. Once its done I'll push it. Getting back to the point. There is no API that can handle large buffer allocation and sharing with low-level attribute control for virtual address spaces outside the CPU. I think we've dealt with the attribute issue to death now. Shall we repeat it again? I think the only point of agreement is that all mappings must have compatible attributes, the issue of multiple mappings is still outstanding, as is needing more fine grained control of the attributes of a set of compatible mappings (I still need to digest your examples a little). The DMA API et al. take a CPU centric view of virtual space management, sharing has to be explicitly written and external virtual space management is left up to device driver writers. I think I've also shown that not to be the case with example code. The code behind the DMA API can be changed on a per-device basis (currently on ARM we haven't supported that because no one's asked for it yet) so that it can support multiple IOMMUs even of multiple different types. I'm seeing that now. As I become more familiar with the DMA API the way forward may become more clear to me. I certainly appreciate the time you've spent discussing things and the code examples you've listed. For example, it fairly clear how I can use a scatter list to describe a mapping of big buffers. I can start down this path and see what shakes out. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majord...@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: a href=mailto:d...@kvack.org; em...@kvack.org /a -- To unsubscribe from this list: send the line unsubscribe linux-omap in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC 3/3] mm: iommu: The Virtual Contiguous Memory Manager
On Thu, Jul 22, 2010 at 01:47:36PM +0900, FUJITA Tomonori wrote: On Wed, 21 Jul 2010 20:50:26 -0700 Zach Pfeffer zpfef...@codeaurora.org wrote: On Wed, Jul 14, 2010 at 10:59:43AM +0900, FUJITA Tomonori wrote: On Tue, 13 Jul 2010 10:02:23 +0100 Zach Pfeffer said this new VCM infrastructure can be useful for video4linux. However, I don't think we need 3,000-lines another abstraction layer to solve video4linux's issue nicely. Its only 3000 lines because I haven't converted the code to use function pointers. The main point is adding a new abstraction that don't provide the huge benefit. I disagree. In its current form the API may not be appropriate for inclusion into the kernel, but it provides a common framework for handling a class of problems that have been solved many times in the kernel: large buffer management, IOMMU interoperation and fine grained mapping control. -- To unsubscribe from this list: send the line unsubscribe linux-omap in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC 3/3] mm: iommu: The Virtual Contiguous Memory Manager
On Wed, Jul 14, 2010 at 10:59:43AM +0900, FUJITA Tomonori wrote: On Tue, 13 Jul 2010 10:02:23 +0100 Zach Pfeffer said this new VCM infrastructure can be useful for video4linux. However, I don't think we need 3,000-lines another abstraction layer to solve video4linux's issue nicely. Its only 3000 lines because I haven't converted the code to use function pointers. I can't find any reasonable reasons that we need to merge VCM; seems that the combination of the current APIs (or with some small extensions) can work for the issues that VCM tries to solve. Getting back to the point. There is no API that can handle large buffer allocation and sharing with low-level attribute control for virtual address spaces outside the CPU. At this point if you need to work with big buffers, 1 MB and 16 MB etc, and map those big buffers to non-CPU virtual spaces you need to explicitly carve them out and set up the mappings and sharing by hand. Its reasonable to have an API that can do this especially since IOMMUs are going to become more prevalent. The DMA API et al. take a CPU centric view of virtual space management, sharing has to be explicitly written and external virtual space management is left up to device driver writers. Given a system where each device has an IOMMU or a MMU the whole concept of a scatterlist goes away. The VCM API gets a jump on it. -- To unsubscribe from this list: send the line unsubscribe linux-omap in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC 3/3] mm: iommu: The Virtual Contiguous Memory Manager
On Wed, 21 Jul 2010 20:50:26 -0700 Zach Pfeffer zpfef...@codeaurora.org wrote: On Wed, Jul 14, 2010 at 10:59:43AM +0900, FUJITA Tomonori wrote: On Tue, 13 Jul 2010 10:02:23 +0100 Zach Pfeffer said this new VCM infrastructure can be useful for video4linux. However, I don't think we need 3,000-lines another abstraction layer to solve video4linux's issue nicely. Its only 3000 lines because I haven't converted the code to use function pointers. The main point is adding a new abstraction that don't provide the huge benefit. -- To unsubscribe from this list: send the line unsubscribe linux-omap in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC 3/3] mm: iommu: The Virtual Contiguous Memory Manager
On Mon, Jul 12, 2010 at 10:21:05PM -0700, Zach Pfeffer wrote: Joerg Roedel wrote: The DMA-API already does this with the help of IOMMUs if they are present. What is the benefit of your approach over that? The grist to the DMA-API mill is the opaque scatterlist. Each scatterlist element brings together a physical address and a bus address that may be different. The set of scatterlist elements constitute both the set of physical buffers and the mappings to those buffers. My approach separates these two things into a struct physmem which contains the set of physical buffers and a struct reservation which contains the set of bus addresses (or device addresses). Each element in the struct physmem may be of various lengths (without resorting to chaining). A map call maps the one set to the other. Okay, thats a different concept, where is the benefit? Joerg -- To unsubscribe from this list: send the line unsubscribe linux-omap in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC 3/3] mm: iommu: The Virtual Contiguous Memory Manager
On Wed, Jul 14, 2010 at 09:34:03PM +0200, Joerg Roedel wrote: On Mon, Jul 12, 2010 at 10:21:05PM -0700, Zach Pfeffer wrote: Joerg Roedel wrote: The DMA-API already does this with the help of IOMMUs if they are present. What is the benefit of your approach over that? The grist to the DMA-API mill is the opaque scatterlist. Each scatterlist element brings together a physical address and a bus address that may be different. The set of scatterlist elements constitute both the set of physical buffers and the mappings to those buffers. My approach separates these two things into a struct physmem which contains the set of physical buffers and a struct reservation which contains the set of bus addresses (or device addresses). Each element in the struct physmem may be of various lengths (without resorting to chaining). A map call maps the one set to the other. Okay, thats a different concept, where is the benefit? The benefit is that virtual address space and physical address space are managed independently. This may be useful if you want to reuse the same set of physical buffers, a user simply maps them when they're needed. It also means that different physical memories could be targeted and a virtual allocation could map those memories without worrying about where they were. This whole concept is just a logical extension of the already existing separation between pages and page frames... in fact the separation between physical memory and what is mapped to that memory is fundamental to the Linux kernel. This approach just says that arbitrary long buffers should work the same way. -- To unsubscribe from this list: send the line unsubscribe linux-omap in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC 3/3] mm: iommu: The Virtual Contiguous Memory Manager
On Mon, 12 Jul 2010 22:46:59 -0700 Zach Pfeffer zpfef...@codeaurora.org wrote: Joerg Roedel wrote: On Fri, Jul 02, 2010 at 12:33:51AM -0700, Zach Pfeffer wrote: Daniel Walker wrote: So if we include this code which map implementations could you collapse into this implementations ? Generally , what currently existing code can VCMM help to eliminate? In theory, it can eliminate all code the interoperates between IOMMU, CPU and non-IOMMU based devices and all the mapping code, alignment, mapping attribute and special block size support that's been implemented. Thats a very abstract statement. Can you point to particular code files and give a rough sketch how it could be improved using VCMM? I can. Not to single out a particular subsystem, but the video4linux code contains interoperation code to abstract the difference between sg buffers, vmalloc buffers and physically contiguous buffers. The VCMM is an attempt to provide a framework where these and all the other buffer types can be unified. Why video4linux can't use the DMA API? Doing DMA with vmalloc'ed buffers is a thing that we should avoid (there are some exceptions like xfs though). -- To unsubscribe from this list: send the line unsubscribe linux-omap in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC 3/3] mm: iommu: The Virtual Contiguous Memory Manager
Why video4linux can't use the DMA API? Doing DMA with vmalloc'ed buffers is a thing that we should avoid (there are some exceptions like xfs though). Vmalloc is about the only API for creating virtually linear memory areas. The video stuff really needs that to avoid lots of horrible special cases when doing buffer processing and the like. Pretty much each driver using it has a pair of functions 'rvmalloc' and 'rvfree' so given a proper vmalloc_for_dma() type interface can easily be switched Alan -- To unsubscribe from this list: send the line unsubscribe linux-omap in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC 3/3] mm: iommu: The Virtual Contiguous Memory Manager
On Tue, 13 Jul 2010 09:20:12 +0100 Alan Cox a...@lxorguk.ukuu.org.uk wrote: Why video4linux can't use the DMA API? Doing DMA with vmalloc'ed buffers is a thing that we should avoid (there are some exceptions like xfs though). Vmalloc is about the only API for creating virtually linear memory areas. The video stuff really needs that to avoid lots of horrible special cases when doing buffer processing and the like. Pretty much each driver using it has a pair of functions 'rvmalloc' and 'rvfree' so given a proper vmalloc_for_dma() type interface can easily be switched We already have helper functions for DMA with vmap pages, flush_kernel_vmap_range and invalidate_kernel_vmap_range. I think that the current DMA API with the above helper functions should work well drivers that want virtually linear large memory areas (such as xfs). -- To unsubscribe from this list: send the line unsubscribe linux-omap in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC 3/3] mm: iommu: The Virtual Contiguous Memory Manager
On Tue, 13 Jul 2010 17:30:43 +0900 FUJITA Tomonori fujita.tomon...@lab.ntt.co.jp wrote: On Tue, 13 Jul 2010 09:20:12 +0100 Alan Cox a...@lxorguk.ukuu.org.uk wrote: Why video4linux can't use the DMA API? Doing DMA with vmalloc'ed buffers is a thing that we should avoid (there are some exceptions like xfs though). Vmalloc is about the only API for creating virtually linear memory areas. The video stuff really needs that to avoid lots of horrible special cases when doing buffer processing and the like. Pretty much each driver using it has a pair of functions 'rvmalloc' and 'rvfree' so given a proper vmalloc_for_dma() type interface can easily be switched We already have helper functions for DMA with vmap pages, flush_kernel_vmap_range and invalidate_kernel_vmap_range. I'm not sure they help at all because the DMA user for these pages isn't the video driver - it's the USB layer, and the USB layer isn't specifically aware it is being passed vmap pages. Alan -- To unsubscribe from this list: send the line unsubscribe linux-omap in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC 3/3] mm: iommu: The Virtual Contiguous Memory Manager
On Tue, 13 Jul 2010 09:42:44 +0100 Alan Cox a...@lxorguk.ukuu.org.uk wrote: On Tue, 13 Jul 2010 17:30:43 +0900 FUJITA Tomonori fujita.tomon...@lab.ntt.co.jp wrote: On Tue, 13 Jul 2010 09:20:12 +0100 Alan Cox a...@lxorguk.ukuu.org.uk wrote: Why video4linux can't use the DMA API? Doing DMA with vmalloc'ed buffers is a thing that we should avoid (there are some exceptions like xfs though). Vmalloc is about the only API for creating virtually linear memory areas. The video stuff really needs that to avoid lots of horrible special cases when doing buffer processing and the like. Pretty much each driver using it has a pair of functions 'rvmalloc' and 'rvfree' so given a proper vmalloc_for_dma() type interface can easily be switched We already have helper functions for DMA with vmap pages, flush_kernel_vmap_range and invalidate_kernel_vmap_range. I'm not sure they help at all because the DMA user for these pages isn't the video driver - it's the USB layer, and the USB layer isn't specifically aware it is being passed vmap pages. Drivers can tell the USB layer that these are vmapped buffers? Adding something to struct urb? I might be totally wrong since I don't know anything about the USB layer. -- To unsubscribe from this list: send the line unsubscribe linux-omap in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC 3/3] mm: iommu: The Virtual Contiguous Memory Manager
On Tue, Jul 13, 2010 at 05:45:39PM +0900, FUJITA Tomonori wrote: Drivers can tell the USB layer that these are vmapped buffers? Adding something to struct urb? I might be totally wrong since I don't know anything about the USB layer. With non-DMA coherent aliasing caches, you need to know where the page is mapped into the virtual address space, so you can deal with aliases. You'd need to tell the USB layer about the other mappings of the page which you'd like to be coherent (such as the vmalloc area - and there's also the possible userspace mapping to think about too, but that's a separate issue.) I wonder if we should have had: vmalloc_prepare_dma(void *, size_t, enum dma_direction) vmalloc_finish_dma(void *, size_t, enum dma_direction) rather than flush_kernel_vmap_range and invalidate_kernel_vmap_range, which'd make their use entirely obvious. However, this brings up a question - how does the driver (eg, v4l, xfs) which is preparing the buffer for another driver (eg, usb host, block dev) know that DMA will be performed on the buffer rather than PIO? That's a very relevant question, because for speculatively prefetching CPUs, we need to invalidate caches after a DMA-from-device operation - but if PIO-from-device happened, this would destroy data read from the device. That problem goes away if we decide that PIO drivers must have the same apparant semantics as DMA drivers - in that data must end up beyond the point of DMA coherency (eg, physical page) - but that's been proven to be very hard to achieve, especially with block device drivers. -- To unsubscribe from this list: send the line unsubscribe linux-omap in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC 3/3] mm: iommu: The Virtual Contiguous Memory Manager
On Tue, Jul 13, 2010 at 02:59:08PM +0900, FUJITA Tomonori wrote: On Mon, 12 Jul 2010 22:46:59 -0700 Zach Pfeffer zpfef...@codeaurora.org wrote: Joerg Roedel wrote: On Fri, Jul 02, 2010 at 12:33:51AM -0700, Zach Pfeffer wrote: Daniel Walker wrote: So if we include this code which map implementations could you collapse into this implementations ? Generally , what currently existing code can VCMM help to eliminate? In theory, it can eliminate all code the interoperates between IOMMU, CPU and non-IOMMU based devices and all the mapping code, alignment, mapping attribute and special block size support that's been implemented. Thats a very abstract statement. Can you point to particular code files and give a rough sketch how it could be improved using VCMM? I can. Not to single out a particular subsystem, but the video4linux code contains interoperation code to abstract the difference between sg buffers, vmalloc buffers and physically contiguous buffers. The VCMM is an attempt to provide a framework where these and all the other buffer types can be unified. Why video4linux can't use the DMA API? Doing DMA with vmalloc'ed buffers is a thing that we should avoid (there are some exceptions like xfs though). I'm not sure, but I know that it makes the distinction. From video4linux/videobuf: media/videobuf-dma-sg.h /* Physically scattered */ media/videobuf-vmalloc.h /* vmalloc() buffers*/ media/videobuf-dma-contig.h /* Physically contiguous */ -- To unsubscribe from this list: send the line unsubscribe linux-omap in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC 3/3] mm: iommu: The Virtual Contiguous Memory Manager
On Tue, 13 Jul 2010 10:02:23 +0100 Russell King - ARM Linux li...@arm.linux.org.uk wrote: On Tue, Jul 13, 2010 at 05:45:39PM +0900, FUJITA Tomonori wrote: Drivers can tell the USB layer that these are vmapped buffers? Adding something to struct urb? I might be totally wrong since I don't know anything about the USB layer. With non-DMA coherent aliasing caches, you need to know where the page is mapped into the virtual address space, so you can deal with aliases. You'd need to tell the USB layer about the other mappings of the page which you'd like to be coherent (such as the vmalloc area - and there's also the possible userspace mapping to think about too, but that's a separate issue.) I wonder if we should have had: vmalloc_prepare_dma(void *, size_t, enum dma_direction) vmalloc_finish_dma(void *, size_t, enum dma_direction) rather than flush_kernel_vmap_range and invalidate_kernel_vmap_range, which'd make their use entirely obvious. However, this brings up a question - how does the driver (eg, v4l, xfs) which is preparing the buffer for another driver (eg, usb host, block dev) know that DMA will be performed on the buffer rather than PIO? That's a very relevant question, because for speculatively prefetching CPUs, we need to invalidate caches after a DMA-from-device operation - but if PIO-from-device happened, this would destroy data read from the device. That problem goes away if we decide that PIO drivers must have the same apparant semantics as DMA drivers - in that data must end up beyond the point of DMA coherency (eg, physical page) - but that's been proven to be very hard to achieve, especially with block device drivers. Yeah, the last thing we want to do is converting all the PIO drivers. Seems that we are far from the original discussion (the most of the above topics have been discussed in the past and we kinda agree that we have to do something some time, I guess). Zach Pfeffer said this new VCM infrastructure can be useful for video4linux. However, I don't think we need 3,000-lines another abstraction layer to solve video4linux's issue nicely. I can't find any reasonable reasons that we need to merge VCM; seems that the combination of the current APIs (or with some small extensions) can work for the issues that VCM tries to solve. -- To unsubscribe from this list: send the line unsubscribe linux-omap in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC 3/3] mm: iommu: The Virtual Contiguous Memory Manager
Joerg Roedel wrote: On Fri, Jul 02, 2010 at 12:09:02AM -0700, Zach Pfeffer wrote: Hari Kanigeri wrote: He demonstrated the usage of his code in one of the emails he sent out initially. Did you go over that, and what (or how many) step would you use with the current code to do the same thing? -- So is this patch set adding layers and abstractions to help the User ? If the idea is to share some memory across multiple devices, I guess you can achieve the same by calling the map function provided by iommu module and sharing the mapped address to the 10's or 100's of devices to access the buffers. You would only need a dedicated virtual pool per IOMMU device to manage its virtual memory allocations. Yeah, you can do that. My idea is to get away from explicit addressing and encapsulate the device address to physical address link into a mapping. The DMA-API already does this with the help of IOMMUs if they are present. What is the benefit of your approach over that? The grist to the DMA-API mill is the opaque scatterlist. Each scatterlist element brings together a physical address and a bus address that may be different. The set of scatterlist elements constitute both the set of physical buffers and the mappings to those buffers. My approach separates these two things into a struct physmem which contains the set of physical buffers and a struct reservation which contains the set of bus addresses (or device addresses). Each element in the struct physmem may be of various lengths (without resorting to chaining). A map call maps the one set to the other. -- Sent by an employee of the Qualcomm Innovation Center, Inc. The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum. -- To unsubscribe from this list: send the line unsubscribe linux-omap in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC 3/3] mm: iommu: The Virtual Contiguous Memory Manager
Joerg Roedel wrote: On Thu, Jul 01, 2010 at 03:00:17PM -0700, Zach Pfeffer wrote: Additionally, the current IOMMU interface does not allow users to associate one page table with multiple IOMMUs [...] Thats not true. Multiple IOMMUs are completly handled by the IOMMU drivers. In the case of the IOMMU-API backend drivers this also includes the ability to use page-tables on multiple IOMMUs. Yeah. I see that now. Since the particular topology is run-time configurable all of these use-cases and more can be expressed without pushing the topology into the low-level IOMMU driver. The IOMMU driver has to know about the topology anyway because it needs to know which IOMMU it needs to program for a particular device. Perhaps, but why not create a VCM which can be shared across all mappers in the system? Why bury it in a device driver and make all IOMMU device drivers managed their own virtual spaces? Practically this would entail a minor refactor to the fledging IOMMU interface; adding associate and activate ops. Already, there are ~20 different IOMMU map implementations in the kernel. Had the Linux kernel had the VCMM, many of those implementations could have leveraged the mapping and topology management of a VCMM, while focusing on a few key hardware specific functions (map this physical address, program the page table base register). I partially agree here. All the IOMMU implementations in the Linux kernel have a lot of functionality in common where code could be shared. Work to share code has been done in the past by Fujita Tomonori but there are more places to work on. I am just not sure if a new front-end API is the right way to do this. I don't really think its a new front end API. Its just an API that allows easier mapping manipulation than the current APIs. -- Sent by an employee of the Qualcomm Innovation Center, Inc. The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum. -- To unsubscribe from this list: send the line unsubscribe linux-omap in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC 3/3] mm: iommu: The Virtual Contiguous Memory Manager
Joerg Roedel wrote: On Fri, Jul 02, 2010 at 12:33:51AM -0700, Zach Pfeffer wrote: Daniel Walker wrote: So if we include this code which map implementations could you collapse into this implementations ? Generally , what currently existing code can VCMM help to eliminate? In theory, it can eliminate all code the interoperates between IOMMU, CPU and non-IOMMU based devices and all the mapping code, alignment, mapping attribute and special block size support that's been implemented. Thats a very abstract statement. Can you point to particular code files and give a rough sketch how it could be improved using VCMM? I can. Not to single out a particular subsystem, but the video4linux code contains interoperation code to abstract the difference between sg buffers, vmalloc buffers and physically contiguous buffers. The VCMM is an attempt to provide a framework where these and all the other buffer types can be unified. -- Sent by an employee of the Qualcomm Innovation Center, Inc. The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum. -- To unsubscribe from this list: send the line unsubscribe linux-omap in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC 3/3] mm: iommu: The Virtual Contiguous Memory Manager
Joerg Roedel wrote: On Thu, Jul 01, 2010 at 11:17:34PM -0700, Zach Pfeffer wrote: Andi Kleen wrote: Hmm? dma_map_* does not change any CPU mappings. It only sets up DMA mapping(s). Sure, but I was saying that iommu_map() doesn't just set up the IOMMU mappings, its sets up both the iommu and kernel buffer mappings. What do you mean by kernel buffer mappings? In-kernel mappings whose addresses can be dereferenced. That assumes that all the IOMMUs on the system support the same page table format, right? Actually no. Since the VCMM abstracts a page-table as a Virtual Contiguous Region (VCM) a VCM can be associated with any device, regardless of their individual page table format. The IOMMU-API abstracts a page-table as a domain which can also be associated with any device (behind an iommu). It does, but only by convention. The domain member is just a big catchall void *. It would be more useful to factor out a VCM abstraction, with associated ops. As it stands all IOMMU device driver writters have to re-invent IOMMU virtual address management. -- Sent by an employee of the Qualcomm Innovation Center, Inc. The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum. -- To unsubscribe from this list: send the line unsubscribe linux-omap in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC 3/3] mm: iommu: The Virtual Contiguous Memory Manager
On Fri, Jul 02, 2010 at 12:09:02AM -0700, Zach Pfeffer wrote: Hari Kanigeri wrote: He demonstrated the usage of his code in one of the emails he sent out initially. Did you go over that, and what (or how many) step would you use with the current code to do the same thing? -- So is this patch set adding layers and abstractions to help the User ? If the idea is to share some memory across multiple devices, I guess you can achieve the same by calling the map function provided by iommu module and sharing the mapped address to the 10's or 100's of devices to access the buffers. You would only need a dedicated virtual pool per IOMMU device to manage its virtual memory allocations. Yeah, you can do that. My idea is to get away from explicit addressing and encapsulate the device address to physical address link into a mapping. The DMA-API already does this with the help of IOMMUs if they are present. What is the benefit of your approach over that? Joerg -- To unsubscribe from this list: send the line unsubscribe linux-omap in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC 3/3] mm: iommu: The Virtual Contiguous Memory Manager
On Thu, Jul 01, 2010 at 03:00:17PM -0700, Zach Pfeffer wrote: Additionally, the current IOMMU interface does not allow users to associate one page table with multiple IOMMUs [...] Thats not true. Multiple IOMMUs are completly handled by the IOMMU drivers. In the case of the IOMMU-API backend drivers this also includes the ability to use page-tables on multiple IOMMUs. Since the particular topology is run-time configurable all of these use-cases and more can be expressed without pushing the topology into the low-level IOMMU driver. The IOMMU driver has to know about the topology anyway because it needs to know which IOMMU it needs to program for a particular device. Already, there are ~20 different IOMMU map implementations in the kernel. Had the Linux kernel had the VCMM, many of those implementations could have leveraged the mapping and topology management of a VCMM, while focusing on a few key hardware specific functions (map this physical address, program the page table base register). I partially agree here. All the IOMMU implementations in the Linux kernel have a lot of functionality in common where code could be shared. Work to share code has been done in the past by Fujita Tomonori but there are more places to work on. I am just not sure if a new front-end API is the right way to do this. Joerg -- To unsubscribe from this list: send the line unsubscribe linux-omap in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC 3/3] mm: iommu: The Virtual Contiguous Memory Manager
On Fri, Jul 02, 2010 at 12:33:51AM -0700, Zach Pfeffer wrote: Daniel Walker wrote: So if we include this code which map implementations could you collapse into this implementations ? Generally , what currently existing code can VCMM help to eliminate? In theory, it can eliminate all code the interoperates between IOMMU, CPU and non-IOMMU based devices and all the mapping code, alignment, mapping attribute and special block size support that's been implemented. Thats a very abstract statement. Can you point to particular code files and give a rough sketch how it could be improved using VCMM? Joerg -- To unsubscribe from this list: send the line unsubscribe linux-omap in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC 3/3] mm: iommu: The Virtual Contiguous Memory Manager
On Thu, Jul 01, 2010 at 11:17:34PM -0700, Zach Pfeffer wrote: Andi Kleen wrote: Hmm? dma_map_* does not change any CPU mappings. It only sets up DMA mapping(s). Sure, but I was saying that iommu_map() doesn't just set up the IOMMU mappings, its sets up both the iommu and kernel buffer mappings. What do you mean by kernel buffer mappings? That assumes that all the IOMMUs on the system support the same page table format, right? Actually no. Since the VCMM abstracts a page-table as a Virtual Contiguous Region (VCM) a VCM can be associated with any device, regardless of their individual page table format. The IOMMU-API abstracts a page-table as a domain which can also be associated with any device (behind an iommu). Joerg -- To unsubscribe from this list: send the line unsubscribe linux-omap in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC 3/3] mm: iommu: The Virtual Contiguous Memory Manager
Andi Kleen wrote: The standard Linux approach to such a problem is to write a library that drivers can use for common functionality, not put a middle layer inbetween. Libraries are much more flexible than layers. I've been thinking about this statement. Its very true. I use the genalloc lib which is a great piece of software to manage VCMs (domains in linux/iommu.h parlance?). On our hardware we have 3 things we have to do, use the minimum set of mappings to map a buffer because of the extremely small TLBs in all the IOMMUs we have to support, use special virtual alignments and direct various multimedia flows through certain IOMMUs. To support this we: 1. Use the genalloc lib to allocate virtual space for our IOMMUs, allowing virtual alignment to be specified. 2. Have a maxmunch allocator that manages our own physical pool. I think I may be able to support this using the iommu interface and some util functions. The big thing that's lost is the unified topology management, but as demonstrated that may fall out from a refactor. Anyhow, sounds like a few things to try. Thanks for the feedback so far. I'll do some refactoring and see what's missing. -- Sent by an employee of the Qualcomm Innovation Center, Inc. The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum. -- To unsubscribe from this list: send the line unsubscribe linux-omap in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC 3/3] mm: iommu: The Virtual Contiguous Memory Manager
Andi Kleen wrote: The VCMM provides a more abstract, global view with finer-grained control of each mapping a user wants to create. For instance, the semantics of iommu_map preclude its use in setting up just the IOMMU side of a mapping. With a one-sided map, two IOMMU devices can be Hmm? dma_map_* does not change any CPU mappings. It only sets up DMA mapping(s). Sure, but I was saying that iommu_map() doesn't just set up the IOMMU mappings, its sets up both the iommu and kernel buffer mappings. Additionally, the current IOMMU interface does not allow users to associate one page table with multiple IOMMUs unless the user explicitly That assumes that all the IOMMUs on the system support the same page table format, right? Actually no. Since the VCMM abstracts a page-table as a Virtual Contiguous Region (VCM) a VCM can be associated with any device, regardless of their individual page table format. As I understand your approach would help if you have different IOMMus with an different low level interface, which just happen to have the same pte format. Is that very likely? I would assume if you have lots of copies of the same IOMMU in the system then you could just use a single driver with multiple instances that share some state for all of them. That model would fit in the current interfaces. There's no reason multiple instances couldn't share the same allocation data structure. And if you have lots of truly different IOMMUs then they likely won't be able to share PTEs at the hardware level anyways, because the formats are too different. See VCM's above. The VCMM takes the long view. Its designed for a future in which the number of IOMMUs will go up and the ways in which these IOMMUs are composed will vary from system to system, and may vary at runtime. Already, there are ~20 different IOMMU map implementations in the kernel. Had the Linux kernel had the VCMM, many of those implementations could have leveraged the mapping and topology management of a VCMM, while focusing on a few key hardware specific functions (map this physical address, program the page table base register). The standard Linux approach to such a problem is to write a library that drivers can use for common functionality, not put a middle layer in between. Libraries are much more flexible than layers. That's true up to the, is this middle layer so useful that its worth it point. The VM is a middle layer, you could make the same argument about it, the mapping code isn't too hard, just map in the memory that you need and be done with it. But the VM middle layer provides a clean separation between page frames and pages which turns out to be infinitely useful. The VCMM is built in the same spirit, It says things like, mapping is a global problem, I'm going to abstract entire virtual spaces and allow people arbitrary chuck size allocation, I'm not going to care that my device is physically mapping this buffer and this other device is a virtual, virtual device. That said I'm not sure there's all that much duplicated code anyways. A lot of the code is always IOMMU specific. The only piece which might be shareable is the mapping allocation, but I don't think that's very much of a typical driver In my old pci-gart driver the allocation was all only a few lines of code, although given it was somewhat dumb in this regard because it only managed a small remapping window. I agree that its not a lot of code, and that this layer may be a bit heavy, but I'd like to focus on is a global mapping view useful and if so is something like the graph management that the VCMM provides generally useful. -- Sent by an employee of the Qualcomm Innovation Center, Inc. The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum. -- To unsubscribe from this list: send the line unsubscribe linux-omap in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC 3/3] mm: iommu: The Virtual Contiguous Memory Manager
Hari Kanigeri wrote: He demonstrated the usage of his code in one of the emails he sent out initially. Did you go over that, and what (or how many) step would you use with the current code to do the same thing? -- So is this patch set adding layers and abstractions to help the User ? If the idea is to share some memory across multiple devices, I guess you can achieve the same by calling the map function provided by iommu module and sharing the mapped address to the 10's or 100's of devices to access the buffers. You would only need a dedicated virtual pool per IOMMU device to manage its virtual memory allocations. Yeah, you can do that. My idea is to get away from explicit addressing and encapsulate the device address to physical address link into a mapping. -- Sent by an employee of the Qualcomm Innovation Center, Inc. The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum. -- To unsubscribe from this list: send the line unsubscribe linux-omap in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC 3/3] mm: iommu: The Virtual Contiguous Memory Manager
Hari Kanigeri wrote: The VCMM takes the long view. Its designed for a future in which the number of IOMMUs will go up and the ways in which these IOMMUs are composed will vary from system to system, and may vary at runtime. Already, there are ~20 different IOMMU map implementations in the kernel. Had the Linux kernel had the VCMM, many of those implementations could have leveraged the mapping and topology management of a VCMM, while focusing on a few key hardware specific functions (map this physical address, program the page table base register). -- Sounds good. Did you think of a way to handle the cases where one of the Device that is using the mapped address crashed ? How is the physical address unbacked in this case ? Actually the API takes care of that by design. Since the physical space is managed apart from the mapper the mapper can crash and not affect the physical memory allocation. -- Sent by an employee of the Qualcomm Innovation Center, Inc. The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum. -- To unsubscribe from this list: send the line unsubscribe linux-omap in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC 3/3] mm: iommu: The Virtual Contiguous Memory Manager
Daniel Walker wrote: On Thu, 2010-07-01 at 15:00 -0700, Zach Pfeffer wrote: Additionally, the current IOMMU interface does not allow users to associate one page table with multiple IOMMUs unless the user explicitly wrote a muxed device underneith the IOMMU interface. This also could be done, but would have to be done for every such use case. Since the particular topology is run-time configurable all of these use-cases and more can be expressed without pushing the topology into the low-level IOMMU driver. The VCMM takes the long view. Its designed for a future in which the number of IOMMUs will go up and the ways in which these IOMMUs are composed will vary from system to system, and may vary at runtime. Already, there are ~20 different IOMMU map implementations in the kernel. Had the Linux kernel had the VCMM, many of those implementations could have leveraged the mapping and topology management of a VCMM, while focusing on a few key hardware specific functions (map this physical address, program the page table base register). So if we include this code which map implementations could you collapse into this implementations ? Generally , what currently existing code can VCMM help to eliminate? In theory, it can eliminate all code the interoperates between IOMMU, CPU and non-IOMMU based devices and all the mapping code, alignment, mapping attribute and special block size support that's been implemented. -- Sent by an employee of the Qualcomm Innovation Center, Inc. The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum. -- To unsubscribe from this list: send the line unsubscribe linux-omap in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC 3/3] mm: iommu: The Virtual Contiguous Memory Manager
On Thu, Jul 01, 2010 at 11:17:34PM -0700, Zach Pfeffer wrote: Andi Kleen wrote: The VCMM provides a more abstract, global view with finer-grained control of each mapping a user wants to create. For instance, the semantics of iommu_map preclude its use in setting up just the IOMMU side of a mapping. With a one-sided map, two IOMMU devices can be Hmm? dma_map_* does not change any CPU mappings. It only sets up DMA mapping(s). Sure, but I was saying that iommu_map() doesn't just set up the IOMMU mappings, its sets up both the iommu and kernel buffer mappings. Normally the data is already in the kernel or mappings, so why would you need another CPU mapping too? Sometimes the CPU code has to scatter-gather, but that is considered acceptable (and if it really cannot be rewritten to support sg it's better to have an explicit vmap operation) In general on larger systems with many CPUs changing CPU mappings also gets expensive (because you have to communicate with all cores), and is not a good idea on frequent IO paths. Additionally, the current IOMMU interface does not allow users to associate one page table with multiple IOMMUs unless the user explicitly That assumes that all the IOMMUs on the system support the same page table format, right? Actually no. Since the VCMM abstracts a page-table as a Virtual Contiguous Region (VCM) a VCM can be associated with any device, regardless of their individual page table format. But then there is no real page table sharing, isn't it? The real information should be in the page tables, nowhere else. The standard Linux approach to such a problem is to write a library that drivers can use for common functionality, not put a middle layer in between. Libraries are much more flexible than layers. That's true up to the, is this middle layer so useful that its worth it point. The VM is a middle layer, you could make the same argument about it, the mapping code isn't too hard, just map in the memory that you need and be done with it. But the VM middle layer provides a clean separation between page frames and pages which turns out to be Actually we use both PFNs and struct page *s in many layers up and down, there's not really any layering in that. -Andi -- a...@linux.intel.com -- Speaking for myself only. -- To unsubscribe from this list: send the line unsubscribe linux-omap in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC 3/3] mm: iommu: The Virtual Contiguous Memory Manager
Andi Kleen wrote: On Thu, Jul 01, 2010 at 11:17:34PM -0700, Zach Pfeffer wrote: Andi Kleen wrote: The VCMM provides a more abstract, global view with finer-grained control of each mapping a user wants to create. For instance, the semantics of iommu_map preclude its use in setting up just the IOMMU side of a mapping. With a one-sided map, two IOMMU devices can be Hmm? dma_map_* does not change any CPU mappings. It only sets up DMA mapping(s). Sure, but I was saying that iommu_map() doesn't just set up the IOMMU mappings, its sets up both the iommu and kernel buffer mappings. Normally the data is already in the kernel or mappings, so why would you need another CPU mapping too? Sometimes the CPU code has to scatter-gather, but that is considered acceptable (and if it really cannot be rewritten to support sg it's better to have an explicit vmap operation) In general on larger systems with many CPUs changing CPU mappings also gets expensive (because you have to communicate with all cores), and is not a good idea on frequent IO paths. That's all true, but what a VCMM allows is for these trade-offs to be made by the user for future systems. It may not be too expensive to change the IO path around on future chips or the user may be okay with the performance penalty. A VCMM doesn't enforce a policy on the user, it lets the user make their own policy. Additionally, the current IOMMU interface does not allow users to associate one page table with multiple IOMMUs unless the user explicitly That assumes that all the IOMMUs on the system support the same page table format, right? Actually no. Since the VCMM abstracts a page-table as a Virtual Contiguous Region (VCM) a VCM can be associated with any device, regardless of their individual page table format. But then there is no real page table sharing, isn't it? The real information should be in the page tables, nowhere else. Yeah, and the implementation ensures that it. The VCMM just adds a few fields like start_addr, len and the device. The device still manages the its page-tables. The standard Linux approach to such a problem is to write a library that drivers can use for common functionality, not put a middle layer in between. Libraries are much more flexible than layers. That's true up to the, is this middle layer so useful that its worth it point. The VM is a middle layer, you could make the same argument about it, the mapping code isn't too hard, just map in the memory that you need and be done with it. But the VM middle layer provides a clean separation between page frames and pages which turns out to be Actually we use both PFNs and struct page *s in many layers up and down, there's not really any layering in that. Sure, but the PFNs and the struct page *s are the middle layer. Its just that things haven't been layered on top of them. A VCMM is the higher level abstraction, since it allows the size of the PFs to vary and the consumers of the VCM's to be determined at run-time. -- Sent by an employee of the Qualcomm Innovation Center, Inc. The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum. -- To unsubscribe from this list: send the line unsubscribe linux-omap in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC 3/3] mm: iommu: The Virtual Contiguous Memory Manager
On Tue, 29 Jun 2010 22:55:50 -0700 Zach Pfeffer wrote:
arch/arm/mm/vcm.c | 1901
+
include/linux/vcm.h | 701 +
include/linux/vcm_types.h | 318
3 files changed, 2920 insertions(+), 0 deletions(-)
create mode 100644 arch/arm/mm/vcm.c
create mode 100644 include/linux/vcm.h
create mode 100644 include/linux/vcm_types.h
diff --git a/include/linux/vcm.h b/include/linux/vcm.h
new file mode 100644
index 000..d2a1cd1
--- /dev/null
+++ b/include/linux/vcm.h
@@ -0,0 +1,701 @@
+/* Copyright (c) 2010, Code Aurora Forum. All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are
+ * met:
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * * Redistributions in binary form must reproduce the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer in the documentation and/or other materials provided
+ * with the distribution.
+ * * Neither the name of Code Aurora Forum, Inc. nor the names of its
+ * contributors may be used to endorse or promote products derived
+ * from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED AS IS AND ANY EXPRESS OR IMPLIED
+ * WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NON-INFRINGEMENT
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS
+ * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
+ * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
+ * WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE
+ * OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN
+ * IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ *
+ */
What license (name/type) is this?
+
+#ifndef _VCM_H_
+#define _VCM_H_
+
+/* All undefined types must be defined using platform specific headers */
+
+#include linux/vcm_types.h
+
+/*
+ * Creating, freeing and managing VCMs.
+ *
+ * A VCM region is a virtual space that can be reserved from and
+ * associated with one or more devices. At creation the user can
+ * specify an offset to start addresses and a length of the entire VCM
+ * region. Reservations out of a VCM region are always contiguous.
+ */
+
+/**
+ * vcm_create() - Create a VCM region
+ * @start_addr The starting address of the VCM region.
* @start_addr: text
* @len:text
I.e., need a colon (':') after each param name. (in multiple places)
+ * @len The len of the VCM region. This must be at least
+ * vcm_get_min_page_size() bytes.
+ *
+ * A VCM typically abstracts a page table.
+ *
+ * All functions in this API are passed and return opaque things
+ * because the underlying implementations will vary. The goal
+ * is really graph management. vcm_create() creates the device end
+ * of an edge in the mapping graph.
+ *
+ * The return value is non-zero if a VCM has successfully been
+ * created. It will return zero if a VCM region cannot be created or
+ * len is invalid.
+ */
+struct vcm *vcm_create(size_t start_addr, size_t len);
+/**
+ * vcm_get_physmem_from_res() - Return a reservation's physmem_id
+ * @ res_id An existing reservation of interest.
* @res_id: text
No space between @ and res_id.
+ *
+ * The return value will be non-zero on success, otherwise it will be:
+ * -EINVAL res is invalid
+ * -ENOMEM res is unbacked
+ */
+struct physmem *vcm_get_physmem_from_res(struct res *res_id);
diff --git a/include/linux/vcm_types.h b/include/linux/vcm_types.h
new file mode 100644
index 000..2cc4770
--- /dev/null
+++ b/include/linux/vcm_types.h
@@ -0,0 +1,318 @@
+/**
+ * enum memtarget_t - A logical location in a VCM.
+ *
+ * VCM_START Indicates the start of a VCM_REGION.
This is not quite kernel-doc notation (as indicated by the beginning /**).
Please see Documentation/kernel-doc-nano-HOWTO.txt for details, or ask me
if you need help with it.
+ */
+enum memtarget_t {
+ VCM_START
+};
+
+
+/**
+ * enum memtype_t - A logical location in a VCM.
not quite kernel-doc notation...
+ *
+ * VCM_MEMTYPE_0 Generic memory type 0
+ * VCM_MEMTYPE_1 Generic memory type 1
+ * VCM_MEMTYPE_2 Generic memory type 2
+ *
+ * A memtype encapsulates a platform specific memory arrangement. The
+ * memtype needn't refer to a single type of memory, it can refer to a
+ * set of memories that can back a reservation.
+ *
+ */
Re: [RFC 3/3] mm: iommu: The Virtual Contiguous Memory Manager
What license (name/type) is this? IANAL, but AFAIK standard wisdom is that disclaimer in the documentation and/or other materials provided is generally not acceptable for Linux because it's an excessive burden for all distributors. Also for me it's still quite unclear why we would want this code at all... It doesn't seem to do anything you couldn't do with the existing interfaces. -Andi -- To unsubscribe from this list: send the line unsubscribe linux-omap in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC 3/3] mm: iommu: The Virtual Contiguous Memory Manager
On Thu, 2010-07-01 at 20:02 +0200, Andi Kleen wrote: What license (name/type) is this? IANAL, but AFAIK standard wisdom is that disclaimer in the documentation and/or other materials provided is generally not acceptable for Linux because it's an excessive burden for all distributors. It's the BSD license .. Also for me it's still quite unclear why we would want this code at all... It doesn't seem to do anything you couldn't do with the existing interfaces. I don't know all that much about what Zach's done here, but from what he's said so far it looks like this help to manage lots of IOMMUs on a single system.. On x86 it seems like there's not all that many IOMMUs in comparison .. Zach mentioned 10 to 100 IOMMUs .. Daniel -- Sent by a consultant of the Qualcomm Innovation Center, Inc. The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum. -- To unsubscribe from this list: send the line unsubscribe linux-omap in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC 3/3] mm: iommu: The Virtual Contiguous Memory Manager
On Thu, Jul 01, 2010 at 12:28:23PM -0700, Daniel Walker wrote: On Thu, 2010-07-01 at 20:02 +0200, Andi Kleen wrote: What license (name/type) is this? IANAL, but AFAIK standard wisdom is that disclaimer in the documentation and/or other materials provided is generally not acceptable for Linux because it's an excessive burden for all distributors. It's the BSD license .. It's the old version of the BSD license that noone uses anymore because of its ] problems: it's really a unreasonable burden to include hundreds or thousands of attributions for every contributor in every printed manual you ship. The BSDs have all switched to the Clause 2 (without this one) because of this. Also for me it's still quite unclear why we would want this code at all... It doesn't seem to do anything you couldn't do with the existing interfaces. I don't know all that much about what Zach's done here, but from what he's said so far it looks like this help to manage lots of IOMMUs on a single system.. On x86 it seems like there's not all that many IOMMUs in comparison .. Zach mentioned 10 to 100 IOMMUs .. The current code can manage multiple IOMMUs fine. -Andi -- a...@linux.intel.com -- Speaking for myself only. -- To unsubscribe from this list: send the line unsubscribe linux-omap in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC 3/3] mm: iommu: The Virtual Contiguous Memory Manager
On Thu, 2010-07-01 at 21:38 +0200, Andi Kleen wrote: Also for me it's still quite unclear why we would want this code at all... It doesn't seem to do anything you couldn't do with the existing interfaces. I don't know all that much about what Zach's done here, but from what he's said so far it looks like this help to manage lots of IOMMUs on a single system.. On x86 it seems like there's not all that many IOMMUs in comparison .. Zach mentioned 10 to 100 IOMMUs .. The current code can manage multiple IOMMUs fine. He demonstrated the usage of his code in one of the emails he sent out initially. Did you go over that, and what (or how many) step would you use with the current code to do the same thing? Daniel -- Sent by a consultant of the Qualcomm Innovation Center, Inc. The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum. -- To unsubscribe from this list: send the line unsubscribe linux-omap in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC 3/3] mm: iommu: The Virtual Contiguous Memory Manager
Randy, On Thu, 1 Jul 2010, Randy Dunlap wrote: + * @start_addr The starting address of the VCM region. + * @lenThe len of the VCM region. This must be at least + * vcm_min() bytes. and missing lots of struct members here. If some of them are private, you can use: /* private: */ ... /* public: */ comments in the struct below and then don't add the private ones to the kernel-doc notation above. To avoid wasting space in structures, it makes sense to place fields smaller than the alignment width together in the structure definition. If one were to do this and follow your proposal, some structures may need multiple private and public comments, which seems undesirable. The alternative, wasting memory, also seems undesirable. Perhaps you might have a proposal for a way to resolve this? - Paul -- To unsubscribe from this list: send the line unsubscribe linux-omap in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC 3/3] mm: iommu: The Virtual Contiguous Memory Manager
On 07/01/10 13:59, Paul Walmsley wrote: Randy, On Thu, 1 Jul 2010, Randy Dunlap wrote: + * @start_addr The starting address of the VCM region. + * @lenThe len of the VCM region. This must be at least + * vcm_min() bytes. and missing lots of struct members here. If some of them are private, you can use: /* private: */ ... /* public: */ comments in the struct below and then don't add the private ones to the kernel-doc notation above. To avoid wasting space in structures, it makes sense to place fields smaller than the alignment width together in the structure definition. If one were to do this and follow your proposal, some structures may need multiple private and public comments, which seems undesirable. The alternative, wasting memory, also seems undesirable. Perhaps you might have a proposal for a way to resolve this? I don't know of a really good way. There are a few structs that have multiple private/public entries, and that is OK. Or you can describe all of the entries with kernel-doc notation. Or you can choose not to use kernel-doc notation on some structs. -- ~Randy *** Remember to use Documentation/SubmitChecklist when testing your code *** -- To unsubscribe from this list: send the line unsubscribe linux-omap in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC 3/3] mm: iommu: The Virtual Contiguous Memory Manager
Andi Kleen wrote: Also for me it's still quite unclear why we would want this code at all... It doesn't seem to do anything you couldn't do with the existing interfaces. I don't know all that much about what Zach's done here, but from what he's said so far it looks like this help to manage lots of IOMMUs on a single system.. On x86 it seems like there's not all that many IOMMUs in comparison .. Zach mentioned 10 to 100 IOMMUs .. The current code can manage multiple IOMMUs fine. That's fair. The current code does manage multiple IOMMUs without issue for a static map topology. Its core function 'map' maps a physical chunk of some size into a IOMMU's address space and the kernel's address space for some domain. The VCMM provides a more abstract, global view with finer-grained control of each mapping a user wants to create. For instance, the symantics of iommu_map preclude its use in setting up just the IOMMU side of a mapping. With a one-sided map, two IOMMU devices can be pointed to the same physical memory without mapping that same memory into the kernel's address space. Additionally, the current IOMMU interface does not allow users to associate one page table with multiple IOMMUs unless the user explicitly wrote a muxed device underneith the IOMMU interface. This also could be done, but would have to be done for every such use case. Since the particular topology is run-time configurable all of these use-cases and more can be expressed without pushing the topology into the low-level IOMMU driver. The VCMM takes the long view. Its designed for a future in which the number of IOMMUs will go up and the ways in which these IOMMUs are composed will vary from system to system, and may vary at runtime. Already, there are ~20 different IOMMU map implementations in the kernel. Had the Linux kernel had the VCMM, many of those implementations could have leveraged the mapping and topology management of a VCMM, while focusing on a few key hardware specific functions (map this physical address, program the page table base register). -- Sent by an employee of the Qualcomm Innovation Center, Inc. The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum. -- To unsubscribe from this list: send the line unsubscribe linux-omap in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC 3/3] mm: iommu: The Virtual Contiguous Memory Manager
On Thu, 2010-07-01 at 15:00 -0700, Zach Pfeffer wrote: Additionally, the current IOMMU interface does not allow users to associate one page table with multiple IOMMUs unless the user explicitly wrote a muxed device underneith the IOMMU interface. This also could be done, but would have to be done for every such use case. Since the particular topology is run-time configurable all of these use-cases and more can be expressed without pushing the topology into the low-level IOMMU driver. The VCMM takes the long view. Its designed for a future in which the number of IOMMUs will go up and the ways in which these IOMMUs are composed will vary from system to system, and may vary at runtime. Already, there are ~20 different IOMMU map implementations in the kernel. Had the Linux kernel had the VCMM, many of those implementations could have leveraged the mapping and topology management of a VCMM, while focusing on a few key hardware specific functions (map this physical address, program the page table base register). So if we include this code which map implementations could you collapse into this implementations ? Generally , what currently existing code can VCMM help to eliminate? Daniel -- Sent by a consultant of the Qualcomm Innovation Center, Inc. The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum. -- To unsubscribe from this list: send the line unsubscribe linux-omap in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC 3/3] mm: iommu: The Virtual Contiguous Memory Manager
He demonstrated the usage of his code in one of the emails he sent out initially. Did you go over that, and what (or how many) step would you use with the current code to do the same thing? -- So is this patch set adding layers and abstractions to help the User ? If the idea is to share some memory across multiple devices, I guess you can achieve the same by calling the map function provided by iommu module and sharing the mapped address to the 10's or 100's of devices to access the buffers. You would only need a dedicated virtual pool per IOMMU device to manage its virtual memory allocations. Hari -- To unsubscribe from this list: send the line unsubscribe linux-omap in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC 3/3] mm: iommu: The Virtual Contiguous Memory Manager
The VCMM takes the long view. Its designed for a future in which the number of IOMMUs will go up and the ways in which these IOMMUs are composed will vary from system to system, and may vary at runtime. Already, there are ~20 different IOMMU map implementations in the kernel. Had the Linux kernel had the VCMM, many of those implementations could have leveraged the mapping and topology management of a VCMM, while focusing on a few key hardware specific functions (map this physical address, program the page table base register). -- Sounds good. Did you think of a way to handle the cases where one of the Device that is using the mapped address crashed ? How is the physical address unbacked in this case ? Hari -- To unsubscribe from this list: send the line unsubscribe linux-omap in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC 3/3] mm: iommu: The Virtual Contiguous Memory Manager
The VCMM provides a more abstract, global view with finer-grained control of each mapping a user wants to create. For instance, the symantics of iommu_map preclude its use in setting up just the IOMMU side of a mapping. With a one-sided map, two IOMMU devices can be Hmm? dma_map_* does not change any CPU mappings. It only sets up DMA mapping(s). Additionally, the current IOMMU interface does not allow users to associate one page table with multiple IOMMUs unless the user explicitly That assumes that all the IOMMUs on the system support the same page table format, right? As I understand your approach would help if you have different IOMMus with an different low level interface, which just happen to have the same pte format. Is that very likely? I would assume if you have lots of copies of the same IOMMU in the system then you could just use a single driver with multiple instances that share some state for all of them. That model would fit in the current interfaces. There's no reason multiple instances couldn't share the same allocation data structure. And if you have lots of truly different IOMMUs then they likely won't be able to share PTEs at the hardware level anyways, because the formats are too different. The VCMM takes the long view. Its designed for a future in which the number of IOMMUs will go up and the ways in which these IOMMUs are composed will vary from system to system, and may vary at runtime. Already, there are ~20 different IOMMU map implementations in the kernel. Had the Linux kernel had the VCMM, many of those implementations could have leveraged the mapping and topology management of a VCMM, while focusing on a few key hardware specific functions (map this physical address, program the page table base register). The standard Linux approach to such a problem is to write a library that drivers can use for common functionality, not put a middle layer inbetween. Libraries are much more flexible than layers. That said I'm not sure there's all that much duplicated code anyways. A lot of the code is always IOMMU specific. The only piece which might be shareable is the mapping allocation, but I don't think that's very much of a typical driver In my old pci-gart driver the allocation was all only a few lines of code, although given it was somewhat dumb in this regard because it only managed a small remapping window. -Andi -- a...@linux.intel.com -- Speaking for myself only. -- To unsubscribe from this list: send the line unsubscribe linux-omap in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
