On Fri, Feb 15, 2019 at 11:07:22AM -0500, Michael Labriola wrote: > > > But the latter text seems to agree with that. So what is the actual > > > problem that started this discussion? > > > > > > > See https://lists.xen.org/archives/html/xen-devel/2019-02/threads.html#00818 > > I believe the actual problem is either: > > 1) The radeon/amdgpu drivers are calling ttm_populate_and_map_pages() > which *should* work on a Xen PV host, but doesn't and needs to be > fixed. > > or > > 2) The radeon/amdgpu drivers are calling ttm_populate_and_map_pages() > which *cannot* work in Xen, and they should go back to calling > ttm_dma_populate() in that case. > > I'm having a hard time figuring out which of those is correct.
I think the answer is neither 1 or 2 (or a bit of both). ttm_populate_and_map_pages uses dma_map_page to map GPU memory, and from what I can tell from your report potentially lots of it. So this does "properly" use the DMA API for some amount of "properly". The problem here is that ttm_populate_and_map_pages first allocates memory and then maps it in a way where it bounce buffers a lot, leading to a swiotlb buffer exaustion, as seen in your report. ttm_dma_populate also sort of "properly" uses the DMA API in that it uses the dma_alloc_coherent allocator. The benefit of that allocator is that is always returns addressable memory without exhausing the swiotlb buffer. The dowside of ttm_dma_populate / dma_alloc_coherent is that on architectures where PCIe is not cache coherent it pointlessly up other resources, as coherent DMA memory can be a very finite resource there. So for a short term fix forcing the dma_alloc_coherent route on Xen/x86 is the right thing. On Xen/arm and Xen/arm64 is might already be problemeatic due to the explanation above unfortunately. The real fix is to finally get broadly available non-coherent device memory allocator into mainlaine and with ttm_populate_and_map_pages to use it. Note that for non-coherent devices it seems like ttm_populate_and_map_pages also seems to miss proper ownership management but that is another issue. My proposal for such an allocator here: https://lwn.net/Articles/774429/ unfortunately doesn't seem to go anywhere as the DRM folks generally seem to prefer bolting band-aid ontop of band-aid instead of actually fixing such problems. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel