> -----Original Message----- > From: Joerg Roedel [mailto:j...@8bytes.org] > Sent: Wednesday, June 28, 2017 4:37 AM > To: Jan Vesely; Deucher, Alexander > Cc: Lendacky, Thomas; Nath, Arindam; Craig Stein; iommu@lists.linux- > foundation.org; Duran, Leo; Suthikulpanit, Suravee > Subject: Re: [PATCH v1 3/3] iommu/amd: Optimize the IOMMU queue flush > > [Adding Alex Deucher] > > Hey Alex, > > On Tue, Jun 27, 2017 at 12:24:35PM -0400, Jan Vesely wrote: > > On Mon, 2017-06-26 at 14:14 +0200, Joerg Roedel wrote: > > > > How does that 'dGPU goes to sleep' work? Do you put it to sleep > manually > > > via sysfs or something? Or is that something that amdgpu does on its > > > own? > > > > AMD folks should be able to provide more details. afaik, the driver > > uses ACPI methods to power on/off the device. Driver routines wake the > > device up before accessing it and there is a timeout to turn it off > > after few seconds of inactivity. > > > > > > > > It looks like the GPU just switches the ATS unit off when it goes to > > > sleep and doesn't answer the invalidation anymore, which explains the > > > completion-wait timeouts. > > > > Both MMIO regs and PCIe config regs are turned off so it would not > > surprise me if all PCIe requests were ignored by the device in off > > state. it should be possible to request device wake up before > > invalidating the relevant IOMMU domain. I'll leave to more > > knowledgeable ppl to judge whether it's a good idea (we can also > > postpone such invalidations until the device is woken by other means) > > Can you maybe sched some light on how the sleep-mode of the GPUs work? > Is it initiated by the GPU driver or from somewhere else? In the case > discussed here it looks like the ATS unit of the GPU is switched of, > causing IOTLB invalidation timeouts on the IOMMU side. > > If that is the case we might need some sort of dma-api extension so that > the GPU driver can tell the iommu driver that the device is going to be > quiet.
I assume you are talking about Hybrid/PowerXpress laptops where the dGPU can be powered down dynamically? That is done via the runtime pm subsystem in the kernel. We register several callbacks with that, and then it takes care of the power down auto timers and such. The actual mechanism to power down the GPU varies for platform to platform (platform specific ACPI methods on early systems, D3cold on newer ones). Alex _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
