Op 09-07-2020 om 14:33 schreef Daniel Vetter: > Comes up every few years, gets somewhat tedious to discuss, let's > write this down once and for all. > > What I'm not sure about is whether the text should be more explicit in > flat out mandating the amdkfd eviction fences for long running compute > workloads or workloads where userspace fencing is allowed. > > v2: Now with dot graph! > > v3: Typo (Dave Airlie)
For first 5 patches, and patch 16, 17: Reviewed-by: Maarten Lankhorst <maarten.lankho...@linux.intel.com> > Acked-by: Christian König <christian.koe...@amd.com> > Acked-by: Daniel Stone <dani...@collabora.com> > Cc: Jesse Natalie <jenat...@microsoft.com> > Cc: Steve Pronovost <spron...@microsoft.com> > Cc: Jason Ekstrand <ja...@jlekstrand.net> > Cc: Felix Kuehling <felix.kuehl...@amd.com> > Cc: Mika Kuoppala <mika.kuopp...@intel.com> > Cc: Thomas Hellstrom <thomas.hellst...@intel.com> > Cc: linux-me...@vger.kernel.org > Cc: linaro-mm-...@lists.linaro.org > Cc: linux-r...@vger.kernel.org > Cc: amd-...@lists.freedesktop.org > Cc: intel-...@lists.freedesktop.org > Cc: Chris Wilson <ch...@chris-wilson.co.uk> > Cc: Maarten Lankhorst <maarten.lankho...@linux.intel.com> > Cc: Christian König <christian.koe...@amd.com> > Signed-off-by: Daniel Vetter <daniel.vet...@intel.com> > --- > Documentation/driver-api/dma-buf.rst | 70 ++++++++++++++++++++++++++++ > 1 file changed, 70 insertions(+) > > diff --git a/Documentation/driver-api/dma-buf.rst > b/Documentation/driver-api/dma-buf.rst > index f8f6decde359..100bfd227265 100644 > --- a/Documentation/driver-api/dma-buf.rst > +++ b/Documentation/driver-api/dma-buf.rst > @@ -178,3 +178,73 @@ DMA Fence uABI/Sync File > .. kernel-doc:: include/linux/sync_file.h > :internal: > > +Indefinite DMA Fences > +~~~~~~~~~~~~~~~~~~~~ > + > +At various times &dma_fence with an indefinite time until dma_fence_wait() > +finishes have been proposed. Examples include: > + > +* Future fences, used in HWC1 to signal when a buffer isn't used by the > display > + any longer, and created with the screen update that makes the buffer > visible. > + The time this fence completes is entirely under userspace's control. > + > +* Proxy fences, proposed to handle &drm_syncobj for which the fence has not > yet > + been set. Used to asynchronously delay command submission. > + > +* Userspace fences or gpu futexes, fine-grained locking within a command > buffer > + that userspace uses for synchronization across engines or with the CPU, > which > + are then imported as a DMA fence for integration into existing winsys > + protocols. > + > +* Long-running compute command buffers, while still using traditional end of > + batch DMA fences for memory management instead of context preemption DMA > + fences which get reattached when the compute job is rescheduled. > + > +Common to all these schemes is that userspace controls the dependencies of > these > +fences and controls when they fire. Mixing indefinite fences with normal > +in-kernel DMA fences does not work, even when a fallback timeout is included > to > +protect against malicious userspace: > + > +* Only the kernel knows about all DMA fence dependencies, userspace is not > aware > + of dependencies injected due to memory management or scheduler decisions. > + > +* Only userspace knows about all dependencies in indefinite fences and when > + exactly they will complete, the kernel has no visibility. > + > +Furthermore the kernel has to be able to hold up userspace command submission > +for memory management needs, which means we must support indefinite fences > being > +dependent upon DMA fences. If the kernel also support indefinite fences in > the > +kernel like a DMA fence, like any of the above proposal would, there is the > +potential for deadlocks. > + > +.. kernel-render:: DOT > + :alt: Indefinite Fencing Dependency Cycle > + :caption: Indefinite Fencing Dependency Cycle > + > + digraph "Fencing Cycle" { > + node [shape=box bgcolor=grey style=filled] > + kernel [label="Kernel DMA Fences"] > + userspace [label="userspace controlled fences"] > + kernel -> userspace [label="memory management"] > + userspace -> kernel [label="Future fence, fence proxy, ..."] > + > + { rank=same; kernel userspace } > + } > + > +This means that the kernel might accidentally create deadlocks > +through memory management dependencies which userspace is unaware of, which > +randomly hangs workloads until the timeout kicks in. Workloads, which from > +userspace's perspective, do not contain a deadlock. In such a mixed fencing > +architecture there is no single entity with knowledge of all dependencies. > +Thefore preventing such deadlocks from within the kernel is not possible. > + > +The only solution to avoid dependencies loops is by not allowing indefinite > +fences in the kernel. This means: > + > +* No future fences, proxy fences or userspace fences imported as DMA fences, > + with or without a timeout. > + > +* No DMA fences that signal end of batchbuffer for command submission where > + userspace is allowed to use userspace fencing or long running compute > + workloads. This also means no implicit fencing for shared buffers in these > + cases. _______________________________________________ dri-devel mailing list dri-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/dri-devel