Re: [Intel-gfx] [PATCH v15 16/23] drm/shmem-helper: Use kref for vmap_use_count
On 8/28/23 13:00, Boris Brezillon wrote: > On Sun, 27 Aug 2023 20:54:42 +0300 > Dmitry Osipenko wrote: > >> Use kref helper for vmap_use_count to make refcounting consistent with >> pages_use_count and pages_pin_count that use kref. This will allow to >> optimize unlocked vmappings by skipping reservation locking if refcnt > 1. > > The core is taking the resv lock before calling ->v[un]map(), so > switching to a kref sounds a bit premature/useless, unless there are > plans to delegate the locking to the drivers. The only thing it brings > is standard overflow/underflow checks. Not really sure it's worth > transitioning to a kref for this field until we have a real use case. The overflow checks worth transitioning. I'll mention them in the commit message for v16. -- Best regards, Dmitry
Re: [Intel-gfx] [PATCH v15 17/23] drm/shmem-helper: Add and use drm_gem_shmem_resv_assert_held() helper
On 8/29/23 10:29, Boris Brezillon wrote:
> On Tue, 29 Aug 2023 05:34:23 +0300
> Dmitry Osipenko wrote:
>
>> On 8/28/23 13:12, Boris Brezillon wrote:
>>> On Sun, 27 Aug 2023 20:54:43 +0300
>>> Dmitry Osipenko wrote:
>>>
>>>> In a preparation of adding drm-shmem memory shrinker, move all reservation
>>>> locking lockdep checks to use new drm_gem_shmem_resv_assert_held() that
>>>> will resolve spurious lockdep warning about wrong locking order vs
>>>> fs_reclam code paths during freeing of shmem GEM, where lockdep isn't
>>>> aware that it's impossible to have locking contention with the fs_reclam
>>>> at this special time.
>>>>
>>>> Signed-off-by: Dmitry Osipenko
>>>> ---
>>>> drivers/gpu/drm/drm_gem_shmem_helper.c | 37 +-
>>>> 1 file changed, 25 insertions(+), 12 deletions(-)
>>>>
>>>> diff --git a/drivers/gpu/drm/drm_gem_shmem_helper.c
>>>> b/drivers/gpu/drm/drm_gem_shmem_helper.c
>>>> index d96fee3d6166..ca5da976aafa 100644
>>>> --- a/drivers/gpu/drm/drm_gem_shmem_helper.c
>>>> +++ b/drivers/gpu/drm/drm_gem_shmem_helper.c
>>>> @@ -128,6 +128,23 @@ struct drm_gem_shmem_object
>>>> *drm_gem_shmem_create(struct drm_device *dev, size_t
>>>> }
>>>> EXPORT_SYMBOL_GPL(drm_gem_shmem_create);
>>>>
>>>> +static void drm_gem_shmem_resv_assert_held(struct drm_gem_shmem_object
>>>> *shmem)
>>>> +{
>>>> + /*
>>>> + * Destroying the object is a special case.. drm_gem_shmem_free()
>>>> + * calls many things that WARN_ON if the obj lock is not held. But
>>>> + * acquiring the obj lock in drm_gem_shmem_free() can cause a locking
>>>> + * order inversion between reservation_ww_class_mutex and fs_reclaim.
>>>> + *
>>>> + * This deadlock is not actually possible, because no one should
>>>> + * be already holding the lock when drm_gem_shmem_free() is called.
>>>> + * Unfortunately lockdep is not aware of this detail. So when the
>>>> + * refcount drops to zero, we pretend it is already locked.
>>>> + */
>>>> + if (kref_read(>base.refcount))
>>>> + drm_gem_shmem_resv_assert_held(shmem);
>>>> +}
>>>> +
>>>> /**
>>>> * drm_gem_shmem_free - Free resources associated with a shmem GEM object
>>>> * @shmem: shmem GEM object to free
>>>> @@ -142,8 +159,6 @@ void drm_gem_shmem_free(struct drm_gem_shmem_object
>>>> *shmem)
>>>>if (obj->import_attach) {
>>>>drm_prime_gem_destroy(obj, shmem->sgt);
>>>>} else if (!shmem->imported_sgt) {
>>>> - dma_resv_lock(shmem->base.resv, NULL);
>>>> -
>>>>drm_WARN_ON(obj->dev, kref_read(>vmap_use_count));
>>>>
>>>>if (shmem->sgt) {
>>>> @@ -156,8 +171,6 @@ void drm_gem_shmem_free(struct drm_gem_shmem_object
>>>> *shmem)
>>>>drm_gem_shmem_put_pages_locked(shmem);
>>>
>>> AFAICT, drm_gem_shmem_put_pages_locked() is the only function that's
>>> called in the free path and would complain about resv-lock not being
>>> held. I think I'd feel more comfortable if we were adding a
>>> drm_gem_shmem_free_pages() function that did everything
>>> drm_gem_shmem_put_pages_locked() does except for the lock_held() check
>>> and the refcount dec, and have it called here (and in
>>> drm_gem_shmem_put_pages_locked()). This way we can keep using
>>> dma_resv_assert_held() instead of having our own variant.
>>
>> It's not only drm_gem_shmem_free_pages(), but any drm-shmem function
>> that drivers may use in the GEM's freeing callback.
>>
>> For example, panfrost_gem_free_object() may unpin shmem BO and then do
>> drm_gem_shmem_free().
>
> Is this really a valid use case? If the GEM refcount dropped to zero,
> we should certainly not have pages_pin_count > 0 (thinking of vmap-ed
> buffers that might disappear while kernel still has a pointer to the
> CPU-mapped area). The only reason we have this
> drm_gem_shmem_put_pages_locked() in drm_gem_shmem_free() is because of
> this implicit ref hold by the sgt, and IMHO, we should be stricter and
> check that pages_use_count == 1 when sgt != NULL and pages_use_count ==
> 0 otherwise.
>
> I actually think it's a good thing to try and catch any attempt to call
> functions trying lock the resv in a path they're not supposed to. At
> least we can decide whether these actions are valid or not in this
> context, and provide dedicated helpers for the free path if they are.
To me it's a valid use-case. I was going to do it for the virtio-gpu
driver for a specific BO type that should be permanently pinned in
memory. So I made the BO pinned in the virto_gpu's bo_create() and
unpinned it from the virtio-gpu's gem->free(), this is a perfectly valid
case to me. Though, in the end I switched to another approach that
doesn't require to do the pinning in the virtio-gpu driver.
For now we can do it as you suggested, to use custom put_pages() in the
shmem_free() since neither of drivers need that. Let's try that.
--
Best regards,
Dmitry
Re: [Intel-gfx] [PATCH v15 02/23] drm/shmem-helper: Use flag for tracking page count bumped by get_pages_sgt()
On 8/28/23 13:55, Boris Brezillon wrote: > On Sun, 27 Aug 2023 20:54:28 +0300 > Dmitry Osipenko wrote: > >> Use separate flag for tracking page count bumped by shmem->sgt to avoid >> imbalanced page counter during of drm_gem_shmem_free() time. It's fragile >> to assume that populated shmem->pages at a freeing time means that the >> count was bumped by drm_gem_shmem_get_pages_sgt(), using a flag removes >> the ambiguity. >> >> Signed-off-by: Dmitry Osipenko >> --- >> drivers/gpu/drm/drm_gem_shmem_helper.c | 3 ++- >> drivers/gpu/drm/lima/lima_gem.c| 1 + >> include/drm/drm_gem_shmem_helper.h | 7 +++ >> 3 files changed, 10 insertions(+), 1 deletion(-) >> >> diff --git a/drivers/gpu/drm/drm_gem_shmem_helper.c >> b/drivers/gpu/drm/drm_gem_shmem_helper.c >> index 78d9cf2355a5..db20b9123891 100644 >> --- a/drivers/gpu/drm/drm_gem_shmem_helper.c >> +++ b/drivers/gpu/drm/drm_gem_shmem_helper.c >> @@ -152,7 +152,7 @@ void drm_gem_shmem_free(struct drm_gem_shmem_object >> *shmem) >> sg_free_table(shmem->sgt); >> kfree(shmem->sgt); >> } >> -if (shmem->pages) >> +if (shmem->got_sgt) >> drm_gem_shmem_put_pages(shmem); > > Can't we just move this drm_gem_shmem_put_pages() call in the > if (shmem->sgt) block? As you've seen in patch #1, the shmem->sgt may belong to imported dmabuf and pages aren't referenced in this case. I agree that the freeing code is confusing. The flags make it a better, not ideal. Though, the flags+comments solution is good enough to me. Please let me know if you have more suggestions, otherwise I'll add comment to the code and keep this patch for v16. BTW, I realized that the new flag wasn't placed properly in the Lima driver, causing unbalanced page count in the error path. Will correct it in v16. -- Best regards, Dmitry
Re: [Intel-gfx] [PATCH v15 01/23] drm/shmem-helper: Fix UAF in error path when freeing SGT of imported GEM
On 8/28/23 14:16, Boris Brezillon wrote:
> On Sun, 27 Aug 2023 20:54:27 +0300
> Dmitry Osipenko wrote:
>
>> Freeing drm-shmem GEM right after creating it using
>> drm_gem_shmem_prime_import_sg_table() frees SGT of the imported dma-buf
>> and then dma-buf frees this SGT second time.
>>
>> The v3d_prime_import_sg_table() is example of a error code path where
>> dma-buf's SGT is freed by drm-shmem and then it's freed second time by
>> dma_buf_unmap_attachment() in drm_gem_prime_import_dev().
>>
>> Add drm-shmem GEM flag telling that this is imported SGT shall not be
>> treated as own SGT, fixing the use-after-free bug.
>>
>> Cc: sta...@vger.kernel.org
>> Fixes: 2194a63a818d ("drm: Add library for shmem backed GEM objects")
>> Signed-off-by: Dmitry Osipenko
>> ---
>> drivers/gpu/drm/drm_gem_shmem_helper.c | 3 ++-
>> include/drm/drm_gem_shmem_helper.h | 7 +++
>> 2 files changed, 9 insertions(+), 1 deletion(-)
>>
>> diff --git a/drivers/gpu/drm/drm_gem_shmem_helper.c
>> b/drivers/gpu/drm/drm_gem_shmem_helper.c
>> index a783d2245599..78d9cf2355a5 100644
>> --- a/drivers/gpu/drm/drm_gem_shmem_helper.c
>> +++ b/drivers/gpu/drm/drm_gem_shmem_helper.c
>> @@ -141,7 +141,7 @@ void drm_gem_shmem_free(struct drm_gem_shmem_object
>> *shmem)
>>
>> if (obj->import_attach) {
>> drm_prime_gem_destroy(obj, shmem->sgt);
>> -} else {
>> +} else if (!shmem->imported_sgt) {
>> dma_resv_lock(shmem->base.resv, NULL);
>>
>> drm_WARN_ON(obj->dev, shmem->vmap_use_count);
>> @@ -758,6 +758,7 @@ drm_gem_shmem_prime_import_sg_table(struct drm_device
>> *dev,
>> return ERR_CAST(shmem);
>>
>> shmem->sgt = sgt;
>> +shmem->imported_sgt = true;
>
>
> I feel like adding more fields that can be used to do the is_imported()
> check is going to be even more confusing. Can we instead have
>
> /* drm_gem_shmem_prime_import_sg_table() can be called from a
>* driver specific ->import_sg_table() implementations that
>* have extra failable initialization steps. Assign
>* drm_gem_object::import_attach here (even though it's
>* assigned in drm_gem_prime_import_dev()), so we don't end up
>* with driver error paths calling drm_gem_shmem_free() with an
>* imported sg_table assigned to drm_gem_shmem_object::sgt and
>* drm_gem_object::import_attach left uninitialized.
>*/
> shmem->base.import_attach = attach;
>
> here?
AFAICT, this is not going to work because obj->import_attach will be
released by drm_prime core by the time drm_gem_shmem_free() is invoked
and drm_gem_shmem_free() uses obj->import_attach as well. I'll keep this
patch around unless there will be other suggestions. To me the flag is
good enough, I'll add a clarifying comment to the code in v16.
--
Best regards,
Dmitry
Re: [Intel-gfx] [PATCH v15 00/23] Add generic memory shrinker to VirtIO-GPU and Panfrost DRM drivers
On 8/28/23 18:24, Helen Mae Koike Fornazier wrote: > On Monday, August 28, 2023 11:37 -03, "Helen Mae Koike Fornazier" > wrote: > >> On Sunday, August 27, 2023 14:54 -03, Dmitry Osipenko >> wrote: >> >>> This series: >>> >>> 1. Adds common drm-shmem memory shrinker >>> 2. Enables shrinker for VirtIO-GPU driver >>> 3. Switches Panfrost driver to the common shrinker >> >> Hi Dmitry, >> >> Would you mind testing with drm-ci? We virt-io tests there and it would be >> really great to get your feedback of it. >> >> https://cgit.freedesktop.org/drm/drm/log/?h=topic/drm-ci > > sorry, I forgot that you also need this patchset: > https://lists.freedesktop.org/archives/dri-devel/2023-August/420063.html > to enable virtio_gpu test job. > > Thanks again. > Helen > >> >> You need to merge your changes with the above tree. >> To configure it, you just need to have a tree on gitlab.freedesktop.org, >> go to the settings and change the CI/CD configuration file from >> .gitlab-ci.yml >> to drivers/gpu/drm/ci/gitlab-ci.yml, and you can start a pipeline >> on your branch. >> >> at the time of this writting, gitlab.freedesktop.org is under maintenance, >> but it should be back soon. Thanks, Helen. I'll give it a try for the next version -- Best regards, Dmitry
Re: [Intel-gfx] [PATCH v15 17/23] drm/shmem-helper: Add and use drm_gem_shmem_resv_assert_held() helper
On 8/28/23 13:12, Boris Brezillon wrote:
> On Sun, 27 Aug 2023 20:54:43 +0300
> Dmitry Osipenko wrote:
>
>> In a preparation of adding drm-shmem memory shrinker, move all reservation
>> locking lockdep checks to use new drm_gem_shmem_resv_assert_held() that
>> will resolve spurious lockdep warning about wrong locking order vs
>> fs_reclam code paths during freeing of shmem GEM, where lockdep isn't
>> aware that it's impossible to have locking contention with the fs_reclam
>> at this special time.
>>
>> Signed-off-by: Dmitry Osipenko
>> ---
>> drivers/gpu/drm/drm_gem_shmem_helper.c | 37 +-
>> 1 file changed, 25 insertions(+), 12 deletions(-)
>>
>> diff --git a/drivers/gpu/drm/drm_gem_shmem_helper.c
>> b/drivers/gpu/drm/drm_gem_shmem_helper.c
>> index d96fee3d6166..ca5da976aafa 100644
>> --- a/drivers/gpu/drm/drm_gem_shmem_helper.c
>> +++ b/drivers/gpu/drm/drm_gem_shmem_helper.c
>> @@ -128,6 +128,23 @@ struct drm_gem_shmem_object
>> *drm_gem_shmem_create(struct drm_device *dev, size_t
>> }
>> EXPORT_SYMBOL_GPL(drm_gem_shmem_create);
>>
>> +static void drm_gem_shmem_resv_assert_held(struct drm_gem_shmem_object
>> *shmem)
>> +{
>> +/*
>> + * Destroying the object is a special case.. drm_gem_shmem_free()
>> + * calls many things that WARN_ON if the obj lock is not held. But
>> + * acquiring the obj lock in drm_gem_shmem_free() can cause a locking
>> + * order inversion between reservation_ww_class_mutex and fs_reclaim.
>> + *
>> + * This deadlock is not actually possible, because no one should
>> + * be already holding the lock when drm_gem_shmem_free() is called.
>> + * Unfortunately lockdep is not aware of this detail. So when the
>> + * refcount drops to zero, we pretend it is already locked.
>> + */
>> +if (kref_read(>base.refcount))
>> +drm_gem_shmem_resv_assert_held(shmem);
>> +}
>> +
>> /**
>> * drm_gem_shmem_free - Free resources associated with a shmem GEM object
>> * @shmem: shmem GEM object to free
>> @@ -142,8 +159,6 @@ void drm_gem_shmem_free(struct drm_gem_shmem_object
>> *shmem)
>> if (obj->import_attach) {
>> drm_prime_gem_destroy(obj, shmem->sgt);
>> } else if (!shmem->imported_sgt) {
>> -dma_resv_lock(shmem->base.resv, NULL);
>> -
>> drm_WARN_ON(obj->dev, kref_read(>vmap_use_count));
>>
>> if (shmem->sgt) {
>> @@ -156,8 +171,6 @@ void drm_gem_shmem_free(struct drm_gem_shmem_object
>> *shmem)
>> drm_gem_shmem_put_pages_locked(shmem);
>
> AFAICT, drm_gem_shmem_put_pages_locked() is the only function that's
> called in the free path and would complain about resv-lock not being
> held. I think I'd feel more comfortable if we were adding a
> drm_gem_shmem_free_pages() function that did everything
> drm_gem_shmem_put_pages_locked() does except for the lock_held() check
> and the refcount dec, and have it called here (and in
> drm_gem_shmem_put_pages_locked()). This way we can keep using
> dma_resv_assert_held() instead of having our own variant.
It's not only drm_gem_shmem_free_pages(), but any drm-shmem function
that drivers may use in the GEM's freeing callback.
For example, panfrost_gem_free_object() may unpin shmem BO and then do
drm_gem_shmem_free().
--
Best regards,
Dmitry
Re: [Intel-gfx] [PATCH v15 12/23] drm/shmem-helper: Add and use pages_pin_count
On 8/28/23 14:46, Boris Brezillon wrote:
> On Sun, 27 Aug 2023 20:54:38 +0300
> Dmitry Osipenko wrote:
>
>> Add separate pages_pin_count for tracking of whether drm-shmem pages are
>> moveable or not. With the addition of memory shrinker support to drm-shmem,
>> the pages_use_count will no longer determine whether pages are hard-pinned
>> in memory, but whether pages exit and are soft-pinned (and could be swapped
>> out). The pages_pin_count > 1 will hard-pin pages in memory.
>>
>> Suggested-by: Boris Brezillon
>> Signed-off-by: Dmitry Osipenko
>> ---
>> drivers/gpu/drm/drm_gem_shmem_helper.c | 22 +-
>> include/drm/drm_gem_shmem_helper.h | 10 ++
>> 2 files changed, 27 insertions(+), 5 deletions(-)
>>
>> diff --git a/drivers/gpu/drm/drm_gem_shmem_helper.c
>> b/drivers/gpu/drm/drm_gem_shmem_helper.c
>> index d545d3d227d7..1a7e5c332fd8 100644
>> --- a/drivers/gpu/drm/drm_gem_shmem_helper.c
>> +++ b/drivers/gpu/drm/drm_gem_shmem_helper.c
>> @@ -234,14 +234,22 @@ static int drm_gem_shmem_pin_locked(struct
>> drm_gem_shmem_object *shmem)
>>
>> dma_resv_assert_held(shmem->base.resv);
>>
>> +if (kref_get_unless_zero(>pages_pin_count))
>> +return 0;
>> +
>> ret = drm_gem_shmem_get_pages_locked(shmem);
>> +if (!ret)
>> +kref_init(>pages_pin_count);
>>
>> return ret;
>> }
>>
>> -static void drm_gem_shmem_unpin_locked(struct drm_gem_shmem_object *shmem)
>> +static void drm_gem_shmem_kref_unpin_pages(struct kref *kref)
>> {
>> -dma_resv_assert_held(shmem->base.resv);
>> +struct drm_gem_shmem_object *shmem;
>> +
>> +shmem = container_of(kref, struct drm_gem_shmem_object,
>> + pages_pin_count);
>>
>> drm_gem_shmem_put_pages_locked(shmem);
>> }
>> @@ -263,6 +271,9 @@ int drm_gem_shmem_pin(struct drm_gem_shmem_object *shmem)
>>
>> drm_WARN_ON(obj->dev, obj->import_attach);
>>
>> +if (kref_get_unless_zero(>pages_pin_count))
>> +return 0;
>> +
>> ret = dma_resv_lock_interruptible(shmem->base.resv, NULL);
>> if (ret)
>> return ret;
>> @@ -286,9 +297,10 @@ void drm_gem_shmem_unpin(struct drm_gem_shmem_object
>> *shmem)
>>
>> drm_WARN_ON(obj->dev, obj->import_attach);
>>
>> -dma_resv_lock(shmem->base.resv, NULL);
>> -drm_gem_shmem_unpin_locked(shmem);
>> -dma_resv_unlock(shmem->base.resv);
>> +if (kref_put_dma_resv(>pages_pin_count,
>> + drm_gem_shmem_kref_unpin_pages,
>> + obj->resv, NULL))
>> +dma_resv_unlock(obj->resv);
>> }
>> EXPORT_SYMBOL_GPL(drm_gem_shmem_unpin);
>>
>> diff --git a/include/drm/drm_gem_shmem_helper.h
>> b/include/drm/drm_gem_shmem_helper.h
>> index ec2d8b24e3cf..afb7cd671e2a 100644
>> --- a/include/drm/drm_gem_shmem_helper.h
>> +++ b/include/drm/drm_gem_shmem_helper.h
>> @@ -39,6 +39,16 @@ struct drm_gem_shmem_object {
>> */
>> unsigned int pages_use_count;
>>
>> +/**
>> + * @pages_pin_count:
>> + *
>> + * Reference count on the pinned pages table.
>> + * The pages allowed to be evicted and purged by memory
>> + * shrinker only when the count is zero, otherwise pages
>> + * are hard-pinned in memory.
>> + */
>> +struct kref pages_pin_count;
>
> I know it's tempting to use kref for the pages use/pin count, but I'm
> wondering if we wouldn't be better using a refcount_t, which provides
> overflow/underflow protection while still letting us control how we
> want to handle the locking for 0 <-> 1 transitions. By doing that, we
> avoid introducing core locking changes that might be more
> controversial/longer to get accepted. Besides, I suspect the resulting
> code (the one using a refcount_t) won't be more verbose/complicated (no
> release functions needed if you don't use kref_put(), which makes
> things closer to what we have right now).
Alright, let's try to use refcount_t since Christian also doesn't like kref
--
Best regards,
Dmitry
Re: [Intel-gfx] [PATCH v15 10/23] locking/refcount, kref: Add kref_put_ww_mutex()
On 8/28/23 12:26, Boris Brezillon wrote:
> On Sun, 27 Aug 2023 20:54:36 +0300
> Dmitry Osipenko wrote:
>
>> Introduce kref_put_ww_mutex() helper that will handle the wait-wound
>> mutex auto-locking on kref_put(). This helper is wanted by DRM drivers
>> that extensively use dma-reservation locking which in turns uses ww-mutex.
>>
>> Signed-off-by: Dmitry Osipenko
>> ---
>> include/linux/kref.h | 12
>> include/linux/refcount.h | 5 +
>> lib/refcount.c | 34 ++
>> 3 files changed, 51 insertions(+)
>>
>> diff --git a/include/linux/kref.h b/include/linux/kref.h
>> index d32e21a2538c..b2d8dc6e9ae0 100644
>> --- a/include/linux/kref.h
>> +++ b/include/linux/kref.h
>> @@ -90,6 +90,18 @@ static inline int kref_put_lock(struct kref *kref,
>> return 0;
>> }
>>
>> +static inline int kref_put_ww_mutex(struct kref *kref,
>> +void (*release)(struct kref *kref),
>> +struct ww_mutex *lock,
>> +struct ww_acquire_ctx *ctx)
>> +{
>> +if (refcount_dec_and_ww_mutex_lock(>refcount, lock, ctx)) {
>> +release(kref);
>> +return 1;
>> +}
>> +return 0;
>> +}
>> +
>> /**
>> * kref_get_unless_zero - Increment refcount for object unless it is zero.
>> * @kref: object.
>> diff --git a/include/linux/refcount.h b/include/linux/refcount.h
>> index a62fcca97486..be9ad272bc77 100644
>> --- a/include/linux/refcount.h
>> +++ b/include/linux/refcount.h
>> @@ -99,6 +99,8 @@
>> #include
>>
>> struct mutex;
>> +struct ww_mutex;
>> +struct ww_acquire_ctx;
>>
>> /**
>> * typedef refcount_t - variant of atomic_t specialized for reference counts
>> @@ -366,4 +368,7 @@ extern __must_check bool
>> refcount_dec_and_lock(refcount_t *r, spinlock_t *lock)
>> extern __must_check bool refcount_dec_and_lock_irqsave(refcount_t *r,
>> spinlock_t *lock,
>> unsigned long *flags)
>> __cond_acquires(lock);
>> +extern __must_check bool refcount_dec_and_ww_mutex_lock(refcount_t *r,
>> +struct ww_mutex *lock,
>> +struct ww_acquire_ctx
>> *ctx) __cond_acquires(>base);
>> #endif /* _LINUX_REFCOUNT_H */
>> diff --git a/lib/refcount.c b/lib/refcount.c
>> index a207a8f22b3c..3f6fd0ceed02 100644
>> --- a/lib/refcount.c
>> +++ b/lib/refcount.c
>> @@ -6,6 +6,7 @@
>> #include
>> #include
>> #include
>> +#include
>> #include
>>
>> #define REFCOUNT_WARN(str) WARN_ONCE(1, "refcount_t: " str ".\n")
>> @@ -184,3 +185,36 @@ bool refcount_dec_and_lock_irqsave(refcount_t *r,
>> spinlock_t *lock,
>> return true;
>> }
>> EXPORT_SYMBOL(refcount_dec_and_lock_irqsave);
>> +
>> +/**
>> + * refcount_dec_and_ww_mutex_lock - return holding ww-mutex if able to
>> + * decrement refcount to 0
>> + * @r: the refcount
>> + * @lock: the ww-mutex to be locked
>> + * @ctx: wait-wound context
>> + *
>> + * Similar to atomic_dec_and_lock(), it will WARN on underflow and fail to
>> + * decrement when saturated at REFCOUNT_SATURATED.
>> + *
>> + * Provides release memory ordering, such that prior loads and stores are
>> done
>> + * before, and provides a control dependency such that free() must come
>> after.
>> + * See the comment on top.
>> + *
>> + * Return: true and hold ww-mutex lock if able to decrement refcount to 0,
>> + * false otherwise
>> + */
>> +bool refcount_dec_and_ww_mutex_lock(refcount_t *r, struct ww_mutex *lock,
>> +struct ww_acquire_ctx *ctx)
>> +{
>> +if (refcount_dec_not_one(r))
>> +return false;
>> +
>> +ww_mutex_lock(lock, ctx);
>
> Unless I'm wrong, ww_mutex_lock() can return -EDEADLK when ctx !=
> NULL, in which case, the lock is not held when it returns. Question is,
> do we really have a use case for ctx != NULL in that kref_put_ww_mutex()
> path. If we need to acquire other ww_locks, this lock, and the other
> locks should have been acquired beforehand, and we can simply call
> kref_put() when we want to release the ref on the resource.
Right, I completely forgot about the deadlocking
--
Best regards,
Dmitry
[Intel-gfx] [PATCH v15 15/23] drm/shmem-helper: Switch drm_gem_shmem_vmap/vunmap to use pin/unpin
The vmapped pages shall be pinned in memory and previously get/put_pages()
were implicitly hard-pinning/unpinning the pages. This will no longer be
the case with addition of memory shrinker because pages_use_count > 0 won't
determine anymore whether pages are hard-pinned (they will be soft-pinned),
while the new pages_pin_count will do the hard-pinning. Switch the
vmap/vunmap() to use pin/unpin() functions in a preparation of addition
of the memory shrinker support to drm-shmem.
Signed-off-by: Dmitry Osipenko
---
drivers/gpu/drm/drm_gem_shmem_helper.c | 13 ++---
include/drm/drm_gem_shmem_helper.h | 2 +-
2 files changed, 11 insertions(+), 4 deletions(-)
diff --git a/drivers/gpu/drm/drm_gem_shmem_helper.c
b/drivers/gpu/drm/drm_gem_shmem_helper.c
index f386289c24fc..17a0177acb5d 100644
--- a/drivers/gpu/drm/drm_gem_shmem_helper.c
+++ b/drivers/gpu/drm/drm_gem_shmem_helper.c
@@ -274,6 +274,13 @@ static void drm_gem_shmem_kref_unpin_pages(struct kref
*kref)
drm_gem_shmem_put_pages_locked(shmem);
}
+static void drm_gem_shmem_unpin_locked(struct drm_gem_shmem_object *shmem)
+{
+ dma_resv_assert_held(shmem->base.resv);
+
+ kref_put(>pages_pin_count, drm_gem_shmem_kref_unpin_pages);
+}
+
/**
* drm_gem_shmem_pin - Pin backing pages for a shmem GEM object
* @shmem: shmem GEM object
@@ -357,7 +364,7 @@ int drm_gem_shmem_vmap_locked(struct drm_gem_shmem_object
*shmem,
return 0;
}
- ret = drm_gem_shmem_get_pages_locked(shmem);
+ ret = drm_gem_shmem_pin_locked(shmem);
if (ret)
goto err_zero_use;
@@ -380,7 +387,7 @@ int drm_gem_shmem_vmap_locked(struct drm_gem_shmem_object
*shmem,
err_put_pages:
if (!obj->import_attach)
- drm_gem_shmem_put_pages_locked(shmem);
+ drm_gem_shmem_unpin_locked(shmem);
err_zero_use:
shmem->vmap_use_count = 0;
@@ -417,7 +424,7 @@ void drm_gem_shmem_vunmap_locked(struct
drm_gem_shmem_object *shmem,
return;
vunmap(shmem->vaddr);
- drm_gem_shmem_put_pages_locked(shmem);
+ drm_gem_shmem_unpin_locked(shmem);
}
shmem->vaddr = NULL;
diff --git a/include/drm/drm_gem_shmem_helper.h
b/include/drm/drm_gem_shmem_helper.h
index a5a3c193cc8f..400ecd63f45f 100644
--- a/include/drm/drm_gem_shmem_helper.h
+++ b/include/drm/drm_gem_shmem_helper.h
@@ -137,7 +137,7 @@ int drm_gem_shmem_madvise_locked(struct
drm_gem_shmem_object *shmem, int madv);
static inline bool drm_gem_shmem_is_purgeable(struct drm_gem_shmem_object
*shmem)
{
return (shmem->madv > 0) &&
- !shmem->vmap_use_count && shmem->sgt &&
+ !kref_read(>pages_pin_count) && shmem->sgt &&
!shmem->base.dma_buf && !shmem->base.import_attach;
}
--
2.41.0
[Intel-gfx] [PATCH v15 21/23] drm/virtio: Attach shmem BOs dynamically
Prepare for addition of memory shrinker support by attaching shmem pages
to host dynamically on first use. The attachment vq command wasn't fenced
and there was no vq kick made in the BO creation code path, hence the
the attachment already was happening dynamically, but implicitly. Making
attachment explicitly dynamic will allow to simplify and reuse more code
when shrinker will be added. The virtio_gpu_object_shmem_init() now works
under held reservation lock, which will be important to have for shrinker.
Acked-by: Gerd Hoffmann
Signed-off-by: Dmitry Osipenko
---
drivers/gpu/drm/virtio/virtgpu_drv.h| 7 +++
drivers/gpu/drm/virtio/virtgpu_gem.c| 26
drivers/gpu/drm/virtio/virtgpu_ioctl.c | 32 ++
drivers/gpu/drm/virtio/virtgpu_object.c | 80 -
drivers/gpu/drm/virtio/virtgpu_submit.c | 15 -
5 files changed, 132 insertions(+), 28 deletions(-)
diff --git a/drivers/gpu/drm/virtio/virtgpu_drv.h
b/drivers/gpu/drm/virtio/virtgpu_drv.h
index 5a4b74b7b318..8c82530eae82 100644
--- a/drivers/gpu/drm/virtio/virtgpu_drv.h
+++ b/drivers/gpu/drm/virtio/virtgpu_drv.h
@@ -89,6 +89,7 @@ struct virtio_gpu_object {
uint32_t hw_res_handle;
bool dumb;
bool created;
+ bool detached;
bool host3d_blob, guest_blob;
uint32_t blob_mem, blob_flags;
@@ -313,6 +314,8 @@ void virtio_gpu_array_put_free(struct
virtio_gpu_object_array *objs);
void virtio_gpu_array_put_free_delayed(struct virtio_gpu_device *vgdev,
struct virtio_gpu_object_array *objs);
void virtio_gpu_array_put_free_work(struct work_struct *work);
+int virtio_gpu_array_prepare(struct virtio_gpu_device *vgdev,
+struct virtio_gpu_object_array *objs);
int virtio_gpu_gem_pin(struct virtio_gpu_object *bo);
void virtio_gpu_gem_unpin(struct virtio_gpu_object *bo);
@@ -458,6 +461,10 @@ int virtio_gpu_object_create(struct virtio_gpu_device
*vgdev,
bool virtio_gpu_is_shmem(struct virtio_gpu_object *bo);
+int virtio_gpu_reattach_shmem_object_locked(struct virtio_gpu_object *bo);
+
+int virtio_gpu_reattach_shmem_object(struct virtio_gpu_object *bo);
+
int virtio_gpu_resource_id_get(struct virtio_gpu_device *vgdev,
uint32_t *resid);
/* virtgpu_prime.c */
diff --git a/drivers/gpu/drm/virtio/virtgpu_gem.c
b/drivers/gpu/drm/virtio/virtgpu_gem.c
index 625c05d625bf..97e67064c97e 100644
--- a/drivers/gpu/drm/virtio/virtgpu_gem.c
+++ b/drivers/gpu/drm/virtio/virtgpu_gem.c
@@ -295,6 +295,26 @@ void virtio_gpu_array_put_free_work(struct work_struct
*work)
spin_unlock(>obj_free_lock);
}
+int virtio_gpu_array_prepare(struct virtio_gpu_device *vgdev,
+struct virtio_gpu_object_array *objs)
+{
+ struct virtio_gpu_object *bo;
+ int ret = 0;
+ u32 i;
+
+ for (i = 0; i < objs->nents; i++) {
+ bo = gem_to_virtio_gpu_obj(objs->objs[i]);
+
+ if (virtio_gpu_is_shmem(bo) && bo->detached) {
+ ret = virtio_gpu_reattach_shmem_object_locked(bo);
+ if (ret)
+ break;
+ }
+ }
+
+ return ret;
+}
+
int virtio_gpu_gem_pin(struct virtio_gpu_object *bo)
{
int err;
@@ -303,6 +323,12 @@ int virtio_gpu_gem_pin(struct virtio_gpu_object *bo)
err = drm_gem_shmem_pin(>base);
if (err)
return err;
+
+ err = virtio_gpu_reattach_shmem_object(bo);
+ if (err) {
+ drm_gem_shmem_unpin(>base);
+ return err;
+ }
}
return 0;
diff --git a/drivers/gpu/drm/virtio/virtgpu_ioctl.c
b/drivers/gpu/drm/virtio/virtgpu_ioctl.c
index b24b11f25197..070c29cea26a 100644
--- a/drivers/gpu/drm/virtio/virtgpu_ioctl.c
+++ b/drivers/gpu/drm/virtio/virtgpu_ioctl.c
@@ -246,6 +246,10 @@ static int virtio_gpu_transfer_from_host_ioctl(struct
drm_device *dev,
if (ret != 0)
goto err_put_free;
+ ret = virtio_gpu_array_prepare(vgdev, objs);
+ if (ret)
+ goto err_unlock;
+
fence = virtio_gpu_fence_alloc(vgdev, vgdev->fence_drv.context, 0);
if (!fence) {
ret = -ENOMEM;
@@ -288,11 +292,25 @@ static int virtio_gpu_transfer_to_host_ioctl(struct
drm_device *dev, void *data,
goto err_put_free;
}
+ ret = virtio_gpu_array_lock_resv(objs);
+ if (ret != 0)
+ goto err_put_free;
+
+ ret = virtio_gpu_array_prepare(vgdev, objs);
+ if (ret)
+ goto err_unlock;
+
+ fence = virtio_gpu_fence_alloc(vgdev, vgdev->fence_drv.context, 0);
+ if (!fence) {
+ ret = -ENOMEM;
+ goto err_unlock;
+ }
+
if (!vgdev->has_virgl_3d) {
[Intel-gfx] [PATCH v15 22/23] drm/virtio: Support memory shrinking
Support generic drm-shmem memory shrinker and add new madvise IOCTL to
the VirtIO-GPU driver. BO cache manager of Mesa driver will mark BOs as
"don't need" using the new IOCTL to let shrinker purge the marked BOs on
OOM, the shrinker will also evict unpurgeable shmem BOs from memory if
guest supports SWAP file or partition.
Acked-by: Gerd Hoffmann
Signed-off-by: Daniel Almeida
Signed-off-by: Dmitry Osipenko
---
drivers/gpu/drm/virtio/virtgpu_drv.h| 13 +-
drivers/gpu/drm/virtio/virtgpu_gem.c| 35 ++
drivers/gpu/drm/virtio/virtgpu_ioctl.c | 25 ++
drivers/gpu/drm/virtio/virtgpu_kms.c| 8
drivers/gpu/drm/virtio/virtgpu_object.c | 61 +
drivers/gpu/drm/virtio/virtgpu_vq.c | 40
include/uapi/drm/virtgpu_drm.h | 14 ++
7 files changed, 195 insertions(+), 1 deletion(-)
diff --git a/drivers/gpu/drm/virtio/virtgpu_drv.h
b/drivers/gpu/drm/virtio/virtgpu_drv.h
index 8c82530eae82..a34da2036221 100644
--- a/drivers/gpu/drm/virtio/virtgpu_drv.h
+++ b/drivers/gpu/drm/virtio/virtgpu_drv.h
@@ -278,7 +278,7 @@ struct virtio_gpu_fpriv {
};
/* virtgpu_ioctl.c */
-#define DRM_VIRTIO_NUM_IOCTLS 12
+#define DRM_VIRTIO_NUM_IOCTLS 13
extern struct drm_ioctl_desc virtio_gpu_ioctls[DRM_VIRTIO_NUM_IOCTLS];
void virtio_gpu_create_context(struct drm_device *dev, struct drm_file *file);
@@ -316,6 +316,8 @@ void virtio_gpu_array_put_free_delayed(struct
virtio_gpu_device *vgdev,
void virtio_gpu_array_put_free_work(struct work_struct *work);
int virtio_gpu_array_prepare(struct virtio_gpu_device *vgdev,
struct virtio_gpu_object_array *objs);
+int virtio_gpu_gem_host_mem_release(struct virtio_gpu_object *bo);
+int virtio_gpu_gem_madvise(struct virtio_gpu_object *obj, int madv);
int virtio_gpu_gem_pin(struct virtio_gpu_object *bo);
void virtio_gpu_gem_unpin(struct virtio_gpu_object *bo);
@@ -329,6 +331,8 @@ void virtio_gpu_cmd_create_resource(struct
virtio_gpu_device *vgdev,
struct virtio_gpu_fence *fence);
void virtio_gpu_cmd_unref_resource(struct virtio_gpu_device *vgdev,
struct virtio_gpu_object *bo);
+int virtio_gpu_cmd_release_resource(struct virtio_gpu_device *vgdev,
+ struct virtio_gpu_object *bo);
void virtio_gpu_cmd_transfer_to_host_2d(struct virtio_gpu_device *vgdev,
uint64_t offset,
uint32_t width, uint32_t height,
@@ -349,6 +353,9 @@ void virtio_gpu_object_attach(struct virtio_gpu_device
*vgdev,
struct virtio_gpu_object *obj,
struct virtio_gpu_mem_entry *ents,
unsigned int nents);
+void virtio_gpu_object_detach(struct virtio_gpu_device *vgdev,
+ struct virtio_gpu_object *obj,
+ struct virtio_gpu_fence *fence);
int virtio_gpu_attach_status_page(struct virtio_gpu_device *vgdev);
int virtio_gpu_detach_status_page(struct virtio_gpu_device *vgdev);
void virtio_gpu_cursor_ping(struct virtio_gpu_device *vgdev,
@@ -499,4 +506,8 @@ void virtio_gpu_vram_unmap_dma_buf(struct device *dev,
int virtio_gpu_execbuffer_ioctl(struct drm_device *dev, void *data,
struct drm_file *file);
+/* virtgpu_gem_shrinker.c */
+int virtio_gpu_gem_shrinker_init(struct virtio_gpu_device *vgdev);
+void virtio_gpu_gem_shrinker_fini(struct virtio_gpu_device *vgdev);
+
#endif
diff --git a/drivers/gpu/drm/virtio/virtgpu_gem.c
b/drivers/gpu/drm/virtio/virtgpu_gem.c
index 97e67064c97e..748f7bbb0e6d 100644
--- a/drivers/gpu/drm/virtio/virtgpu_gem.c
+++ b/drivers/gpu/drm/virtio/virtgpu_gem.c
@@ -147,10 +147,20 @@ void virtio_gpu_gem_object_close(struct drm_gem_object
*obj,
struct virtio_gpu_device *vgdev = obj->dev->dev_private;
struct virtio_gpu_fpriv *vfpriv = file->driver_priv;
struct virtio_gpu_object_array *objs;
+ struct virtio_gpu_object *bo;
if (!vgdev->has_virgl_3d)
return;
+ bo = gem_to_virtio_gpu_obj(obj);
+
+ /*
+* Purged BO was already detached and released, the resource ID
+* is invalid by now.
+*/
+ if (!virtio_gpu_gem_madvise(bo, VIRTGPU_MADV_WILLNEED))
+ return;
+
objs = virtio_gpu_array_alloc(1);
if (!objs)
return;
@@ -315,6 +325,31 @@ int virtio_gpu_array_prepare(struct virtio_gpu_device
*vgdev,
return ret;
}
+int virtio_gpu_gem_madvise(struct virtio_gpu_object *bo, int madv)
+{
+ if (virtio_gpu_is_shmem(bo))
+ return drm_gem_shmem_object_madvise(>base.base, madv);
+
+ return 1;
+}
+
+int virtio_gpu_gem_host_mem_release(struct virtio_gpu_object *bo)
+{
+ struct virtio_gpu_device *vgdev = bo->base.base.dev-&
[Intel-gfx] [PATCH v15 18/23] drm/shmem-helper: Add memory shrinker
Introduce common drm-shmem shrinker for DRM drivers.
To start using drm-shmem shrinker drivers should do the following:
1. Implement evict() callback of GEM object where driver should check
whether object is purgeable or evictable using drm-shmem helpers and
perform the shrinking action
2. Initialize drm-shmem internals using drmm_gem_shmem_init(drm_device),
which will register drm-shmem shrinker
3. Implement madvise IOCTL that will use drm_gem_shmem_madvise()
Signed-off-by: Daniel Almeida
Signed-off-by: Dmitry Osipenko
---
drivers/gpu/drm/drm_gem_shmem_helper.c| 415 +-
.../gpu/drm/panfrost/panfrost_gem_shrinker.c | 9 +-
include/drm/drm_device.h | 10 +-
include/drm/drm_gem_shmem_helper.h| 71 ++-
4 files changed, 474 insertions(+), 31 deletions(-)
diff --git a/drivers/gpu/drm/drm_gem_shmem_helper.c
b/drivers/gpu/drm/drm_gem_shmem_helper.c
index ca5da976aafa..f0f708e0ff00 100644
--- a/drivers/gpu/drm/drm_gem_shmem_helper.c
+++ b/drivers/gpu/drm/drm_gem_shmem_helper.c
@@ -20,6 +20,7 @@
#include
#include
#include
+#include
#include
#include
@@ -88,8 +89,6 @@ __drm_gem_shmem_create(struct drm_device *dev, size_t size,
bool private)
if (ret)
goto err_release;
- INIT_LIST_HEAD(>madv_list);
-
if (!private) {
/*
* Our buffers are kept pinned, so allocating them
@@ -142,7 +141,42 @@ static void drm_gem_shmem_resv_assert_held(struct
drm_gem_shmem_object *shmem)
* refcount drops to zero, we pretend it is already locked.
*/
if (kref_read(>base.refcount))
- drm_gem_shmem_resv_assert_held(shmem);
+ dma_resv_assert_held(shmem->base.resv);
+}
+
+static bool drm_gem_shmem_is_evictable(struct drm_gem_shmem_object *shmem)
+{
+ drm_gem_shmem_resv_assert_held(shmem);
+
+ return (shmem->madv >= 0) && shmem->base.funcs->evict &&
+ kref_read(>pages_use_count) &&
+ !kref_read(>pages_pin_count) &&
+ !shmem->base.dma_buf && !shmem->base.import_attach &&
+ shmem->sgt && !shmem->evicted;
+}
+
+static void
+drm_gem_shmem_update_pages_state_locked(struct drm_gem_shmem_object *shmem)
+{
+ struct drm_gem_object *obj = >base;
+ struct drm_gem_shmem *shmem_mm = obj->dev->shmem_mm;
+ struct drm_gem_shmem_shrinker *shmem_shrinker = _mm->shrinker;
+
+ drm_gem_shmem_resv_assert_held(shmem);
+
+ if (!shmem_shrinker || obj->import_attach)
+ return;
+
+ if (shmem->madv < 0)
+ drm_gem_lru_remove(>base);
+ else if (drm_gem_shmem_is_evictable(shmem) ||
drm_gem_shmem_is_purgeable(shmem))
+ drm_gem_lru_move_tail(_shrinker->lru_evictable,
>base);
+ else if (shmem->evicted)
+ drm_gem_lru_move_tail(_shrinker->lru_evicted,
>base);
+ else if (!shmem->pages)
+ drm_gem_lru_remove(>base);
+ else
+ drm_gem_lru_move_tail(_shrinker->lru_pinned,
>base);
}
/**
@@ -159,6 +193,9 @@ void drm_gem_shmem_free(struct drm_gem_shmem_object *shmem)
if (obj->import_attach) {
drm_prime_gem_destroy(obj, shmem->sgt);
} else if (!shmem->imported_sgt) {
+ /* take out shmem GEM object from the memory shrinker */
+ drm_gem_shmem_madvise_locked(shmem, -1);
+
drm_WARN_ON(obj->dev, kref_read(>vmap_use_count));
if (shmem->sgt) {
@@ -178,15 +215,26 @@ void drm_gem_shmem_free(struct drm_gem_shmem_object
*shmem)
}
EXPORT_SYMBOL_GPL(drm_gem_shmem_free);
-static int drm_gem_shmem_get_pages_locked(struct drm_gem_shmem_object *shmem)
+static int
+drm_gem_shmem_acquire_pages(struct drm_gem_shmem_object *shmem, bool init)
{
struct drm_gem_object *obj = >base;
struct page **pages;
drm_gem_shmem_resv_assert_held(shmem);
- if (kref_get_unless_zero(>pages_use_count))
+ if (shmem->madv < 0) {
+ drm_WARN_ON(obj->dev, shmem->pages);
+ return -ENOMEM;
+ }
+
+ if (shmem->pages) {
+ drm_WARN_ON(obj->dev, !shmem->evicted);
return 0;
+ }
+
+ if (drm_WARN_ON(obj->dev, !(init ^ kref_read(>pages_use_count
+ return -EINVAL;
pages = drm_gem_get_pages(obj);
if (IS_ERR(pages)) {
@@ -207,20 +255,20 @@ static int drm_gem_shmem_get_pages_locked(struct
drm_gem_shmem_object *shmem)
shmem->pages = pages;
- kref_init(>pages_use_count);
-
return 0;
}
-
-static void drm_gem_shmem_kref_release_pages(struct kref *kref)
+static void
+drm_gem_shmem_release_pages_locked(struct drm_gem_shmem_obj
[Intel-gfx] [PATCH v15 23/23] drm/panfrost: Switch to generic memory shrinker
Replace Panfrost's custom memory shrinker with a common drm-shmem
memory shrinker.
Tested-by: Steven Price # Firefly-RK3288
Reviewed-by: Steven Price
Signed-off-by: Dmitry Osipenko
---
drivers/gpu/drm/panfrost/Makefile | 1 -
drivers/gpu/drm/panfrost/panfrost_device.h| 4 -
drivers/gpu/drm/panfrost/panfrost_drv.c | 27 ++--
drivers/gpu/drm/panfrost/panfrost_gem.c | 30 ++--
drivers/gpu/drm/panfrost/panfrost_gem.h | 9 --
.../gpu/drm/panfrost/panfrost_gem_shrinker.c | 129 --
drivers/gpu/drm/panfrost/panfrost_job.c | 18 ++-
include/drm/drm_gem_shmem_helper.h| 7 -
8 files changed, 47 insertions(+), 178 deletions(-)
delete mode 100644 drivers/gpu/drm/panfrost/panfrost_gem_shrinker.c
diff --git a/drivers/gpu/drm/panfrost/Makefile
b/drivers/gpu/drm/panfrost/Makefile
index 7da2b3f02ed9..11622e22cf15 100644
--- a/drivers/gpu/drm/panfrost/Makefile
+++ b/drivers/gpu/drm/panfrost/Makefile
@@ -5,7 +5,6 @@ panfrost-y := \
panfrost_device.o \
panfrost_devfreq.o \
panfrost_gem.o \
- panfrost_gem_shrinker.o \
panfrost_gpu.o \
panfrost_job.o \
panfrost_mmu.o \
diff --git a/drivers/gpu/drm/panfrost/panfrost_device.h
b/drivers/gpu/drm/panfrost/panfrost_device.h
index b0126b9fbadc..dcc2571c092b 100644
--- a/drivers/gpu/drm/panfrost/panfrost_device.h
+++ b/drivers/gpu/drm/panfrost/panfrost_device.h
@@ -116,10 +116,6 @@ struct panfrost_device {
atomic_t pending;
} reset;
- struct mutex shrinker_lock;
- struct list_head shrinker_list;
- struct shrinker shrinker;
-
struct panfrost_devfreq pfdevfreq;
};
diff --git a/drivers/gpu/drm/panfrost/panfrost_drv.c
b/drivers/gpu/drm/panfrost/panfrost_drv.c
index 175443eacead..8cf338c2a03b 100644
--- a/drivers/gpu/drm/panfrost/panfrost_drv.c
+++ b/drivers/gpu/drm/panfrost/panfrost_drv.c
@@ -170,7 +170,6 @@ panfrost_lookup_bos(struct drm_device *dev,
break;
}
- atomic_inc(>gpu_usecount);
job->mappings[i] = mapping;
}
@@ -395,7 +394,6 @@ static int panfrost_ioctl_madvise(struct drm_device *dev,
void *data,
{
struct panfrost_file_priv *priv = file_priv->driver_priv;
struct drm_panfrost_madvise *args = data;
- struct panfrost_device *pfdev = dev->dev_private;
struct drm_gem_object *gem_obj;
struct panfrost_gem_object *bo;
int ret = 0;
@@ -408,11 +406,15 @@ static int panfrost_ioctl_madvise(struct drm_device *dev,
void *data,
bo = to_panfrost_bo(gem_obj);
+ if (bo->is_heap) {
+ args->retained = 1;
+ goto out_put_object;
+ }
+
ret = dma_resv_lock_interruptible(bo->base.base.resv, NULL);
if (ret)
goto out_put_object;
- mutex_lock(>shrinker_lock);
mutex_lock(>mappings.lock);
if (args->madv == PANFROST_MADV_DONTNEED) {
struct panfrost_gem_mapping *first;
@@ -438,17 +440,8 @@ static int panfrost_ioctl_madvise(struct drm_device *dev,
void *data,
args->retained = drm_gem_shmem_madvise_locked(>base, args->madv);
- if (args->retained) {
- if (args->madv == PANFROST_MADV_DONTNEED)
- list_move_tail(>base.madv_list,
- >shrinker_list);
- else if (args->madv == PANFROST_MADV_WILLNEED)
- list_del_init(>base.madv_list);
- }
-
out_unlock_mappings:
mutex_unlock(>mappings.lock);
- mutex_unlock(>shrinker_lock);
dma_resv_unlock(bo->base.base.resv);
out_put_object:
drm_gem_object_put(gem_obj);
@@ -577,9 +570,6 @@ static int panfrost_probe(struct platform_device *pdev)
ddev->dev_private = pfdev;
pfdev->ddev = ddev;
- mutex_init(>shrinker_lock);
- INIT_LIST_HEAD(>shrinker_list);
-
err = panfrost_device_init(pfdev);
if (err) {
if (err != -EPROBE_DEFER)
@@ -601,10 +591,14 @@ static int panfrost_probe(struct platform_device *pdev)
if (err < 0)
goto err_out1;
- panfrost_gem_shrinker_init(ddev);
+ err = drmm_gem_shmem_init(ddev);
+ if (err < 0)
+ goto err_out2;
return 0;
+err_out2:
+ drm_dev_unregister(ddev);
err_out1:
pm_runtime_disable(pfdev->dev);
panfrost_device_fini(pfdev);
@@ -620,7 +614,6 @@ static void panfrost_remove(struct platform_device *pdev)
struct drm_device *ddev = pfdev->ddev;
drm_dev_unregister(ddev);
- panfrost_gem_shrinker_cleanup(ddev);
pm_runtime_get_sync(pfdev->dev);
pm_runtime_disable(pfdev->dev);
diff --git a/drivers/gpu/drm/panfrost/panfrost_gem.c
b/drivers/gpu/drm/panfrost/panfrost_gem.c
index 5
[Intel-gfx] [PATCH v15 14/23] drm/shmem-helper: Add and use lockless drm_gem_shmem_get_pages()
Add lockless drm_gem_shmem_get_pages() helper that skips taking reservation
lock if pages_use_count is non-zero, leveraging from atomicity of the kref
counter. Make drm_gem_shmem_mmap() to utilize the new helper.
Suggested-by: Boris Brezillon
Signed-off-by: Dmitry Osipenko
---
drivers/gpu/drm/drm_gem_shmem_helper.c | 19 +++
1 file changed, 15 insertions(+), 4 deletions(-)
diff --git a/drivers/gpu/drm/drm_gem_shmem_helper.c
b/drivers/gpu/drm/drm_gem_shmem_helper.c
index 5a2e37b3e51d..f386289c24fc 100644
--- a/drivers/gpu/drm/drm_gem_shmem_helper.c
+++ b/drivers/gpu/drm/drm_gem_shmem_helper.c
@@ -234,6 +234,20 @@ void drm_gem_shmem_put_pages_locked(struct
drm_gem_shmem_object *shmem)
}
EXPORT_SYMBOL_GPL(drm_gem_shmem_put_pages_locked);
+static int drm_gem_shmem_get_pages(struct drm_gem_shmem_object *shmem)
+{
+ int ret;
+
+ if (kref_get_unless_zero(>pages_use_count))
+ return 0;
+
+ dma_resv_lock(shmem->base.resv, NULL);
+ ret = drm_gem_shmem_get_pages_locked(shmem);
+ dma_resv_unlock(shmem->base.resv);
+
+ return ret;
+}
+
static int drm_gem_shmem_pin_locked(struct drm_gem_shmem_object *shmem)
{
int ret;
@@ -616,10 +630,7 @@ int drm_gem_shmem_mmap(struct drm_gem_shmem_object *shmem,
struct vm_area_struct
return ret;
}
- dma_resv_lock(shmem->base.resv, NULL);
- ret = drm_gem_shmem_get_pages_locked(shmem);
- dma_resv_unlock(shmem->base.resv);
-
+ ret = drm_gem_shmem_get_pages(shmem);
if (ret)
return ret;
--
2.41.0
[Intel-gfx] [PATCH v15 19/23] drm/shmem-helper: Export drm_gem_shmem_get_pages_sgt_locked()
Export drm_gem_shmem_get_pages_sgt_locked() that will be used by virtio-gpu
shrinker during GEM swap-in operation done under the held reservation lock.
Signed-off-by: Dmitry Osipenko
---
drivers/gpu/drm/drm_gem_shmem_helper.c | 3 ++-
include/drm/drm_gem_shmem_helper.h | 1 +
2 files changed, 3 insertions(+), 1 deletion(-)
diff --git a/drivers/gpu/drm/drm_gem_shmem_helper.c
b/drivers/gpu/drm/drm_gem_shmem_helper.c
index f0f708e0ff00..62958af90383 100644
--- a/drivers/gpu/drm/drm_gem_shmem_helper.c
+++ b/drivers/gpu/drm/drm_gem_shmem_helper.c
@@ -888,7 +888,7 @@ struct sg_table *drm_gem_shmem_get_sg_table(struct
drm_gem_shmem_object *shmem)
}
EXPORT_SYMBOL_GPL(drm_gem_shmem_get_sg_table);
-static struct sg_table *drm_gem_shmem_get_pages_sgt_locked(struct
drm_gem_shmem_object *shmem)
+struct sg_table *drm_gem_shmem_get_pages_sgt_locked(struct
drm_gem_shmem_object *shmem)
{
struct drm_gem_object *obj = >base;
int ret;
@@ -927,6 +927,7 @@ static struct sg_table
*drm_gem_shmem_get_pages_sgt_locked(struct drm_gem_shmem_
drm_gem_shmem_put_pages_locked(shmem);
return ERR_PTR(ret);
}
+EXPORT_SYMBOL_GPL(drm_gem_shmem_get_pages_sgt_locked);
/**
* drm_gem_shmem_get_pages_sgt - Pin pages, dma map them, and return a
diff --git a/include/drm/drm_gem_shmem_helper.h
b/include/drm/drm_gem_shmem_helper.h
index 112dbe5208c0..e10ba533f74d 100644
--- a/include/drm/drm_gem_shmem_helper.h
+++ b/include/drm/drm_gem_shmem_helper.h
@@ -161,6 +161,7 @@ void drm_gem_shmem_purge_locked(struct drm_gem_shmem_object
*shmem);
struct sg_table *drm_gem_shmem_get_sg_table(struct drm_gem_shmem_object
*shmem);
struct sg_table *drm_gem_shmem_get_pages_sgt(struct drm_gem_shmem_object
*shmem);
+struct sg_table *drm_gem_shmem_get_pages_sgt_locked(struct
drm_gem_shmem_object *shmem);
void drm_gem_shmem_print_info(const struct drm_gem_shmem_object *shmem,
struct drm_printer *p, unsigned int indent);
--
2.41.0
[Intel-gfx] [PATCH v15 17/23] drm/shmem-helper: Add and use drm_gem_shmem_resv_assert_held() helper
In a preparation of adding drm-shmem memory shrinker, move all reservation
locking lockdep checks to use new drm_gem_shmem_resv_assert_held() that
will resolve spurious lockdep warning about wrong locking order vs
fs_reclam code paths during freeing of shmem GEM, where lockdep isn't
aware that it's impossible to have locking contention with the fs_reclam
at this special time.
Signed-off-by: Dmitry Osipenko
---
drivers/gpu/drm/drm_gem_shmem_helper.c | 37 +-
1 file changed, 25 insertions(+), 12 deletions(-)
diff --git a/drivers/gpu/drm/drm_gem_shmem_helper.c
b/drivers/gpu/drm/drm_gem_shmem_helper.c
index d96fee3d6166..ca5da976aafa 100644
--- a/drivers/gpu/drm/drm_gem_shmem_helper.c
+++ b/drivers/gpu/drm/drm_gem_shmem_helper.c
@@ -128,6 +128,23 @@ struct drm_gem_shmem_object *drm_gem_shmem_create(struct
drm_device *dev, size_t
}
EXPORT_SYMBOL_GPL(drm_gem_shmem_create);
+static void drm_gem_shmem_resv_assert_held(struct drm_gem_shmem_object *shmem)
+{
+ /*
+* Destroying the object is a special case.. drm_gem_shmem_free()
+* calls many things that WARN_ON if the obj lock is not held. But
+* acquiring the obj lock in drm_gem_shmem_free() can cause a locking
+* order inversion between reservation_ww_class_mutex and fs_reclaim.
+*
+* This deadlock is not actually possible, because no one should
+* be already holding the lock when drm_gem_shmem_free() is called.
+* Unfortunately lockdep is not aware of this detail. So when the
+* refcount drops to zero, we pretend it is already locked.
+*/
+ if (kref_read(>base.refcount))
+ drm_gem_shmem_resv_assert_held(shmem);
+}
+
/**
* drm_gem_shmem_free - Free resources associated with a shmem GEM object
* @shmem: shmem GEM object to free
@@ -142,8 +159,6 @@ void drm_gem_shmem_free(struct drm_gem_shmem_object *shmem)
if (obj->import_attach) {
drm_prime_gem_destroy(obj, shmem->sgt);
} else if (!shmem->imported_sgt) {
- dma_resv_lock(shmem->base.resv, NULL);
-
drm_WARN_ON(obj->dev, kref_read(>vmap_use_count));
if (shmem->sgt) {
@@ -156,8 +171,6 @@ void drm_gem_shmem_free(struct drm_gem_shmem_object *shmem)
drm_gem_shmem_put_pages_locked(shmem);
drm_WARN_ON(obj->dev, kref_read(>pages_use_count));
-
- dma_resv_unlock(shmem->base.resv);
}
drm_gem_object_release(obj);
@@ -170,7 +183,7 @@ static int drm_gem_shmem_get_pages_locked(struct
drm_gem_shmem_object *shmem)
struct drm_gem_object *obj = >base;
struct page **pages;
- dma_resv_assert_held(shmem->base.resv);
+ drm_gem_shmem_resv_assert_held(shmem);
if (kref_get_unless_zero(>pages_use_count))
return 0;
@@ -228,7 +241,7 @@ static void drm_gem_shmem_kref_release_pages(struct kref
*kref)
*/
void drm_gem_shmem_put_pages_locked(struct drm_gem_shmem_object *shmem)
{
- dma_resv_assert_held(shmem->base.resv);
+ drm_gem_shmem_resv_assert_held(shmem);
kref_put(>pages_use_count, drm_gem_shmem_kref_release_pages);
}
@@ -252,7 +265,7 @@ static int drm_gem_shmem_pin_locked(struct
drm_gem_shmem_object *shmem)
{
int ret;
- dma_resv_assert_held(shmem->base.resv);
+ drm_gem_shmem_resv_assert_held(shmem);
if (kref_get_unless_zero(>pages_pin_count))
return 0;
@@ -276,7 +289,7 @@ static void drm_gem_shmem_kref_unpin_pages(struct kref
*kref)
static void drm_gem_shmem_unpin_locked(struct drm_gem_shmem_object *shmem)
{
- dma_resv_assert_held(shmem->base.resv);
+ drm_gem_shmem_resv_assert_held(shmem);
kref_put(>pages_pin_count, drm_gem_shmem_kref_unpin_pages);
}
@@ -357,7 +370,7 @@ int drm_gem_shmem_vmap_locked(struct drm_gem_shmem_object
*shmem,
} else {
pgprot_t prot = PAGE_KERNEL;
- dma_resv_assert_held(shmem->base.resv);
+ drm_gem_shmem_resv_assert_held(shmem);
if (kref_get_unless_zero(>vmap_use_count)) {
iosys_map_set_vaddr(map, shmem->vaddr);
@@ -426,7 +439,7 @@ void drm_gem_shmem_vunmap_locked(struct
drm_gem_shmem_object *shmem,
if (obj->import_attach) {
dma_buf_vunmap(obj->import_attach->dmabuf, map);
} else {
- dma_resv_assert_held(shmem->base.resv);
+ drm_gem_shmem_resv_assert_held(shmem);
kref_put(>vmap_use_count, drm_gem_shmem_kref_vunmap);
}
@@ -462,7 +475,7 @@ drm_gem_shmem_create_with_handle(struct drm_file *file_priv,
*/
int drm_gem_shmem_madvise_locked(struct drm_gem_shmem_object *shmem, int madv)
{
- dma_resv_assert_held(shmem->base.resv);
+ drm_gem_shmem_resv_assert_held(shmem);
[Intel-gfx] [PATCH v15 20/23] drm/virtio: Pin display framebuffer BO
Prepare to addition of memory shrinker support by pinning display
framebuffer BO pages in memory while they are in use by display on host.
Shrinker is free to relocate framebuffer BO pages if it doesn't know that
pages are in use, thus pin the pages to disallow shrinker to move them.
Acked-by: Gerd Hoffmann
Signed-off-by: Dmitry Osipenko
---
drivers/gpu/drm/virtio/virtgpu_drv.h | 2 ++
drivers/gpu/drm/virtio/virtgpu_gem.c | 19 +++
drivers/gpu/drm/virtio/virtgpu_plane.c | 17 +++--
3 files changed, 36 insertions(+), 2 deletions(-)
diff --git a/drivers/gpu/drm/virtio/virtgpu_drv.h
b/drivers/gpu/drm/virtio/virtgpu_drv.h
index 4126c384286b..5a4b74b7b318 100644
--- a/drivers/gpu/drm/virtio/virtgpu_drv.h
+++ b/drivers/gpu/drm/virtio/virtgpu_drv.h
@@ -313,6 +313,8 @@ void virtio_gpu_array_put_free(struct
virtio_gpu_object_array *objs);
void virtio_gpu_array_put_free_delayed(struct virtio_gpu_device *vgdev,
struct virtio_gpu_object_array *objs);
void virtio_gpu_array_put_free_work(struct work_struct *work);
+int virtio_gpu_gem_pin(struct virtio_gpu_object *bo);
+void virtio_gpu_gem_unpin(struct virtio_gpu_object *bo);
/* virtgpu_vq.c */
int virtio_gpu_alloc_vbufs(struct virtio_gpu_device *vgdev);
diff --git a/drivers/gpu/drm/virtio/virtgpu_gem.c
b/drivers/gpu/drm/virtio/virtgpu_gem.c
index 7db48d17ee3a..625c05d625bf 100644
--- a/drivers/gpu/drm/virtio/virtgpu_gem.c
+++ b/drivers/gpu/drm/virtio/virtgpu_gem.c
@@ -294,3 +294,22 @@ void virtio_gpu_array_put_free_work(struct work_struct
*work)
}
spin_unlock(>obj_free_lock);
}
+
+int virtio_gpu_gem_pin(struct virtio_gpu_object *bo)
+{
+ int err;
+
+ if (virtio_gpu_is_shmem(bo)) {
+ err = drm_gem_shmem_pin(>base);
+ if (err)
+ return err;
+ }
+
+ return 0;
+}
+
+void virtio_gpu_gem_unpin(struct virtio_gpu_object *bo)
+{
+ if (virtio_gpu_is_shmem(bo))
+ drm_gem_shmem_unpin(>base);
+}
diff --git a/drivers/gpu/drm/virtio/virtgpu_plane.c
b/drivers/gpu/drm/virtio/virtgpu_plane.c
index a2e045f3a000..def57b01a826 100644
--- a/drivers/gpu/drm/virtio/virtgpu_plane.c
+++ b/drivers/gpu/drm/virtio/virtgpu_plane.c
@@ -238,20 +238,28 @@ static int virtio_gpu_plane_prepare_fb(struct drm_plane
*plane,
struct virtio_gpu_device *vgdev = dev->dev_private;
struct virtio_gpu_framebuffer *vgfb;
struct virtio_gpu_object *bo;
+ int err;
if (!new_state->fb)
return 0;
vgfb = to_virtio_gpu_framebuffer(new_state->fb);
bo = gem_to_virtio_gpu_obj(vgfb->base.obj[0]);
- if (!bo || (plane->type == DRM_PLANE_TYPE_PRIMARY && !bo->guest_blob))
+
+ err = virtio_gpu_gem_pin(bo);
+ if (err)
+ return err;
+
+ if (plane->type == DRM_PLANE_TYPE_PRIMARY && !bo->guest_blob)
return 0;
if (bo->dumb && (plane->state->fb != new_state->fb)) {
vgfb->fence = virtio_gpu_fence_alloc(vgdev,
vgdev->fence_drv.context,
0);
- if (!vgfb->fence)
+ if (!vgfb->fence) {
+ virtio_gpu_gem_unpin(bo);
return -ENOMEM;
+ }
}
return 0;
@@ -261,15 +269,20 @@ static void virtio_gpu_plane_cleanup_fb(struct drm_plane
*plane,
struct drm_plane_state *state)
{
struct virtio_gpu_framebuffer *vgfb;
+ struct virtio_gpu_object *bo;
if (!state->fb)
return;
vgfb = to_virtio_gpu_framebuffer(state->fb);
+ bo = gem_to_virtio_gpu_obj(vgfb->base.obj[0]);
+
if (vgfb->fence) {
dma_fence_put(>fence->f);
vgfb->fence = NULL;
}
+
+ virtio_gpu_gem_unpin(bo);
}
static void virtio_gpu_cursor_plane_update(struct drm_plane *plane,
--
2.41.0
[Intel-gfx] [PATCH v15 08/23] drm/shmem-helper: Refactor locked/unlocked functions
Add locked and remove unlocked postfixes from drm-shmem function names,
making names consistent with the drm/gem core code.
Suggested-by: Boris Brezillon
Signed-off-by: Dmitry Osipenko
---
drivers/gpu/drm/drm_gem_shmem_helper.c| 64 +--
drivers/gpu/drm/lima/lima_gem.c | 8 +--
drivers/gpu/drm/panfrost/panfrost_drv.c | 2 +-
drivers/gpu/drm/panfrost/panfrost_gem.c | 6 +-
.../gpu/drm/panfrost/panfrost_gem_shrinker.c | 2 +-
drivers/gpu/drm/panfrost/panfrost_mmu.c | 2 +-
drivers/gpu/drm/v3d/v3d_bo.c | 4 +-
drivers/gpu/drm/virtio/virtgpu_object.c | 4 +-
include/drm/drm_gem_shmem_helper.h| 36 +--
9 files changed, 64 insertions(+), 64 deletions(-)
diff --git a/drivers/gpu/drm/drm_gem_shmem_helper.c
b/drivers/gpu/drm/drm_gem_shmem_helper.c
index 575704f38808..f053dc511508 100644
--- a/drivers/gpu/drm/drm_gem_shmem_helper.c
+++ b/drivers/gpu/drm/drm_gem_shmem_helper.c
@@ -43,8 +43,8 @@ static const struct drm_gem_object_funcs drm_gem_shmem_funcs
= {
.pin = drm_gem_shmem_object_pin,
.unpin = drm_gem_shmem_object_unpin,
.get_sg_table = drm_gem_shmem_object_get_sg_table,
- .vmap = drm_gem_shmem_object_vmap,
- .vunmap = drm_gem_shmem_object_vunmap,
+ .vmap = drm_gem_shmem_object_vmap_locked,
+ .vunmap = drm_gem_shmem_object_vunmap_locked,
.mmap = drm_gem_shmem_object_mmap,
.vm_ops = _gem_shmem_vm_ops,
};
@@ -153,7 +153,7 @@ void drm_gem_shmem_free(struct drm_gem_shmem_object *shmem)
kfree(shmem->sgt);
}
if (shmem->got_sgt)
- drm_gem_shmem_put_pages(shmem);
+ drm_gem_shmem_put_pages_locked(shmem);
drm_WARN_ON(obj->dev, shmem->pages_use_count);
@@ -165,7 +165,7 @@ void drm_gem_shmem_free(struct drm_gem_shmem_object *shmem)
}
EXPORT_SYMBOL_GPL(drm_gem_shmem_free);
-static int drm_gem_shmem_get_pages(struct drm_gem_shmem_object *shmem)
+static int drm_gem_shmem_get_pages_locked(struct drm_gem_shmem_object *shmem)
{
struct drm_gem_object *obj = >base;
struct page **pages;
@@ -199,12 +199,12 @@ static int drm_gem_shmem_get_pages(struct
drm_gem_shmem_object *shmem)
}
/*
- * drm_gem_shmem_put_pages - Decrease use count on the backing pages for a
shmem GEM object
+ * drm_gem_shmem_put_pages_locked - Decrease use count on the backing pages
for a shmem GEM object
* @shmem: shmem GEM object
*
* This function decreases the use count and puts the backing pages when use
drops to zero.
*/
-void drm_gem_shmem_put_pages(struct drm_gem_shmem_object *shmem)
+void drm_gem_shmem_put_pages_locked(struct drm_gem_shmem_object *shmem)
{
struct drm_gem_object *obj = >base;
@@ -226,7 +226,7 @@ void drm_gem_shmem_put_pages(struct drm_gem_shmem_object
*shmem)
shmem->pages_mark_accessed_on_put);
shmem->pages = NULL;
}
-EXPORT_SYMBOL_GPL(drm_gem_shmem_put_pages);
+EXPORT_SYMBOL_GPL(drm_gem_shmem_put_pages_locked);
static int drm_gem_shmem_pin_locked(struct drm_gem_shmem_object *shmem)
{
@@ -234,7 +234,7 @@ static int drm_gem_shmem_pin_locked(struct
drm_gem_shmem_object *shmem)
dma_resv_assert_held(shmem->base.resv);
- ret = drm_gem_shmem_get_pages(shmem);
+ ret = drm_gem_shmem_get_pages_locked(shmem);
return ret;
}
@@ -243,7 +243,7 @@ static void drm_gem_shmem_unpin_locked(struct
drm_gem_shmem_object *shmem)
{
dma_resv_assert_held(shmem->base.resv);
- drm_gem_shmem_put_pages(shmem);
+ drm_gem_shmem_put_pages_locked(shmem);
}
/**
@@ -293,7 +293,7 @@ void drm_gem_shmem_unpin(struct drm_gem_shmem_object *shmem)
EXPORT_SYMBOL_GPL(drm_gem_shmem_unpin);
/*
- * drm_gem_shmem_vmap - Create a virtual mapping for a shmem GEM object
+ * drm_gem_shmem_vmap_locked - Create a virtual mapping for a shmem GEM object
* @shmem: shmem GEM object
* @map: Returns the kernel virtual address of the SHMEM GEM object's backing
* store.
@@ -302,13 +302,13 @@ EXPORT_SYMBOL_GPL(drm_gem_shmem_unpin);
* exists for the buffer backing the shmem GEM object. It hides the differences
* between dma-buf imported and natively allocated objects.
*
- * Acquired mappings should be cleaned up by calling drm_gem_shmem_vunmap().
+ * Acquired mappings should be cleaned up by calling
drm_gem_shmem_vunmap_locked().
*
* Returns:
* 0 on success or a negative error code on failure.
*/
-int drm_gem_shmem_vmap(struct drm_gem_shmem_object *shmem,
- struct iosys_map *map)
+int drm_gem_shmem_vmap_locked(struct drm_gem_shmem_object *shmem,
+ struct iosys_map *map)
{
struct drm_gem_object *obj = >base;
int ret = 0;
@@ -331,7 +331,7 @@ int drm_gem_shmem_vmap(struct drm_gem_shmem_object *shmem,
[Intel-gfx] [PATCH v15 09/23] drm/shmem-helper: Remove obsoleted is_iomem test
Everything that uses the mapped buffer should by agnostic to is_iomem.
The only reason for the is_iomem test is that we're setting shmem->vaddr
to the returned map->vaddr. Now that the shmem->vaddr code is gone, remove
the obsoleted is_iomem test to clean up the code.
Suggested-by: Thomas Zimmermann
Signed-off-by: Dmitry Osipenko
---
drivers/gpu/drm/drm_gem_shmem_helper.c | 6 --
1 file changed, 6 deletions(-)
diff --git a/drivers/gpu/drm/drm_gem_shmem_helper.c
b/drivers/gpu/drm/drm_gem_shmem_helper.c
index f053dc511508..d545d3d227d7 100644
--- a/drivers/gpu/drm/drm_gem_shmem_helper.c
+++ b/drivers/gpu/drm/drm_gem_shmem_helper.c
@@ -315,12 +315,6 @@ int drm_gem_shmem_vmap_locked(struct drm_gem_shmem_object
*shmem,
if (obj->import_attach) {
ret = dma_buf_vmap(obj->import_attach->dmabuf, map);
- if (!ret) {
- if (drm_WARN_ON(obj->dev, map->is_iomem)) {
- dma_buf_vunmap(obj->import_attach->dmabuf, map);
- return -EIO;
- }
- }
} else {
pgprot_t prot = PAGE_KERNEL;
--
2.41.0
[Intel-gfx] [PATCH v15 16/23] drm/shmem-helper: Use kref for vmap_use_count
Use kref helper for vmap_use_count to make refcounting consistent with
pages_use_count and pages_pin_count that use kref. This will allow to
optimize unlocked vmappings by skipping reservation locking if refcnt > 1.
Suggested-by: Boris Brezillon
Signed-off-by: Dmitry Osipenko
---
drivers/gpu/drm/drm_gem_shmem_helper.c | 37 ++
include/drm/drm_gem_shmem_helper.h | 2 +-
2 files changed, 21 insertions(+), 18 deletions(-)
diff --git a/drivers/gpu/drm/drm_gem_shmem_helper.c
b/drivers/gpu/drm/drm_gem_shmem_helper.c
index 17a0177acb5d..d96fee3d6166 100644
--- a/drivers/gpu/drm/drm_gem_shmem_helper.c
+++ b/drivers/gpu/drm/drm_gem_shmem_helper.c
@@ -144,7 +144,7 @@ void drm_gem_shmem_free(struct drm_gem_shmem_object *shmem)
} else if (!shmem->imported_sgt) {
dma_resv_lock(shmem->base.resv, NULL);
- drm_WARN_ON(obj->dev, shmem->vmap_use_count);
+ drm_WARN_ON(obj->dev, kref_read(>vmap_use_count));
if (shmem->sgt) {
dma_unmap_sgtable(obj->dev->dev, shmem->sgt,
@@ -359,23 +359,25 @@ int drm_gem_shmem_vmap_locked(struct drm_gem_shmem_object
*shmem,
dma_resv_assert_held(shmem->base.resv);
- if (shmem->vmap_use_count++ > 0) {
+ if (kref_get_unless_zero(>vmap_use_count)) {
iosys_map_set_vaddr(map, shmem->vaddr);
return 0;
}
ret = drm_gem_shmem_pin_locked(shmem);
if (ret)
- goto err_zero_use;
+ return ret;
if (shmem->map_wc)
prot = pgprot_writecombine(prot);
shmem->vaddr = vmap(shmem->pages, obj->size >> PAGE_SHIFT,
VM_MAP, prot);
- if (!shmem->vaddr)
+ if (!shmem->vaddr) {
ret = -ENOMEM;
- else
+ } else {
iosys_map_set_vaddr(map, shmem->vaddr);
+ kref_init(>vmap_use_count);
+ }
}
if (ret) {
@@ -388,13 +390,22 @@ int drm_gem_shmem_vmap_locked(struct drm_gem_shmem_object
*shmem,
err_put_pages:
if (!obj->import_attach)
drm_gem_shmem_unpin_locked(shmem);
-err_zero_use:
- shmem->vmap_use_count = 0;
return ret;
}
EXPORT_SYMBOL_GPL(drm_gem_shmem_vmap_locked);
+static void drm_gem_shmem_kref_vunmap(struct kref *kref)
+{
+ struct drm_gem_shmem_object *shmem;
+
+ shmem = container_of(kref, struct drm_gem_shmem_object,
+vmap_use_count);
+
+ vunmap(shmem->vaddr);
+ drm_gem_shmem_unpin_locked(shmem);
+}
+
/*
* drm_gem_shmem_vunmap_locked - Unmap a virtual mapping for a shmem GEM object
* @shmem: shmem GEM object
@@ -416,15 +427,7 @@ void drm_gem_shmem_vunmap_locked(struct
drm_gem_shmem_object *shmem,
dma_buf_vunmap(obj->import_attach->dmabuf, map);
} else {
dma_resv_assert_held(shmem->base.resv);
-
- if (drm_WARN_ON_ONCE(obj->dev, !shmem->vmap_use_count))
- return;
-
- if (--shmem->vmap_use_count > 0)
- return;
-
- vunmap(shmem->vaddr);
- drm_gem_shmem_unpin_locked(shmem);
+ kref_put(>vmap_use_count, drm_gem_shmem_kref_vunmap);
}
shmem->vaddr = NULL;
@@ -663,7 +666,7 @@ void drm_gem_shmem_print_info(const struct
drm_gem_shmem_object *shmem,
return;
drm_printf_indent(p, indent, "pages_use_count=%u\n",
kref_read(>pages_use_count));
- drm_printf_indent(p, indent, "vmap_use_count=%u\n",
shmem->vmap_use_count);
+ drm_printf_indent(p, indent, "vmap_use_count=%u\n",
kref_read(>vmap_use_count));
drm_printf_indent(p, indent, "vaddr=%p\n", shmem->vaddr);
}
EXPORT_SYMBOL_GPL(drm_gem_shmem_print_info);
diff --git a/include/drm/drm_gem_shmem_helper.h
b/include/drm/drm_gem_shmem_helper.h
index 400ecd63f45f..0e0ccd380f66 100644
--- a/include/drm/drm_gem_shmem_helper.h
+++ b/include/drm/drm_gem_shmem_helper.h
@@ -81,7 +81,7 @@ struct drm_gem_shmem_object {
* Reference count on the virtual address.
* The address are un-mapped when the count reaches zero.
*/
- unsigned int vmap_use_count;
+ struct kref vmap_use_count;
/**
* @got_sgt:
--
2.41.0
[Intel-gfx] [PATCH v15 12/23] drm/shmem-helper: Add and use pages_pin_count
Add separate pages_pin_count for tracking of whether drm-shmem pages are
moveable or not. With the addition of memory shrinker support to drm-shmem,
the pages_use_count will no longer determine whether pages are hard-pinned
in memory, but whether pages exit and are soft-pinned (and could be swapped
out). The pages_pin_count > 1 will hard-pin pages in memory.
Suggested-by: Boris Brezillon
Signed-off-by: Dmitry Osipenko
---
drivers/gpu/drm/drm_gem_shmem_helper.c | 22 +-
include/drm/drm_gem_shmem_helper.h | 10 ++
2 files changed, 27 insertions(+), 5 deletions(-)
diff --git a/drivers/gpu/drm/drm_gem_shmem_helper.c
b/drivers/gpu/drm/drm_gem_shmem_helper.c
index d545d3d227d7..1a7e5c332fd8 100644
--- a/drivers/gpu/drm/drm_gem_shmem_helper.c
+++ b/drivers/gpu/drm/drm_gem_shmem_helper.c
@@ -234,14 +234,22 @@ static int drm_gem_shmem_pin_locked(struct
drm_gem_shmem_object *shmem)
dma_resv_assert_held(shmem->base.resv);
+ if (kref_get_unless_zero(>pages_pin_count))
+ return 0;
+
ret = drm_gem_shmem_get_pages_locked(shmem);
+ if (!ret)
+ kref_init(>pages_pin_count);
return ret;
}
-static void drm_gem_shmem_unpin_locked(struct drm_gem_shmem_object *shmem)
+static void drm_gem_shmem_kref_unpin_pages(struct kref *kref)
{
- dma_resv_assert_held(shmem->base.resv);
+ struct drm_gem_shmem_object *shmem;
+
+ shmem = container_of(kref, struct drm_gem_shmem_object,
+pages_pin_count);
drm_gem_shmem_put_pages_locked(shmem);
}
@@ -263,6 +271,9 @@ int drm_gem_shmem_pin(struct drm_gem_shmem_object *shmem)
drm_WARN_ON(obj->dev, obj->import_attach);
+ if (kref_get_unless_zero(>pages_pin_count))
+ return 0;
+
ret = dma_resv_lock_interruptible(shmem->base.resv, NULL);
if (ret)
return ret;
@@ -286,9 +297,10 @@ void drm_gem_shmem_unpin(struct drm_gem_shmem_object
*shmem)
drm_WARN_ON(obj->dev, obj->import_attach);
- dma_resv_lock(shmem->base.resv, NULL);
- drm_gem_shmem_unpin_locked(shmem);
- dma_resv_unlock(shmem->base.resv);
+ if (kref_put_dma_resv(>pages_pin_count,
+ drm_gem_shmem_kref_unpin_pages,
+ obj->resv, NULL))
+ dma_resv_unlock(obj->resv);
}
EXPORT_SYMBOL_GPL(drm_gem_shmem_unpin);
diff --git a/include/drm/drm_gem_shmem_helper.h
b/include/drm/drm_gem_shmem_helper.h
index ec2d8b24e3cf..afb7cd671e2a 100644
--- a/include/drm/drm_gem_shmem_helper.h
+++ b/include/drm/drm_gem_shmem_helper.h
@@ -39,6 +39,16 @@ struct drm_gem_shmem_object {
*/
unsigned int pages_use_count;
+ /**
+* @pages_pin_count:
+*
+* Reference count on the pinned pages table.
+* The pages allowed to be evicted and purged by memory
+* shrinker only when the count is zero, otherwise pages
+* are hard-pinned in memory.
+*/
+ struct kref pages_pin_count;
+
/**
* @madv: State for madvise
*
--
2.41.0
[Intel-gfx] [PATCH v15 04/23] drm/gem: Add _locked postfix to functions that have unlocked counterpart
Add _locked postfix to drm_gem functions that have unlocked counterpart
functions to make GEM functions naming more consistent and intuitive in
regards to the locking requirements.
Suggested-by: Boris Brezillon
Signed-off-by: Dmitry Osipenko
---
drivers/gpu/drm/drm_gem.c | 6 +++---
include/drm/drm_gem.h | 2 +-
2 files changed, 4 insertions(+), 4 deletions(-)
diff --git a/drivers/gpu/drm/drm_gem.c b/drivers/gpu/drm/drm_gem.c
index fae5832bb0bd..8c0268944199 100644
--- a/drivers/gpu/drm/drm_gem.c
+++ b/drivers/gpu/drm/drm_gem.c
@@ -1488,10 +1488,10 @@ drm_gem_lru_scan(struct drm_gem_lru *lru,
EXPORT_SYMBOL(drm_gem_lru_scan);
/**
- * drm_gem_evict - helper to evict backing pages for a GEM object
+ * drm_gem_evict_locked - helper to evict backing pages for a GEM object
* @obj: obj in question
*/
-int drm_gem_evict(struct drm_gem_object *obj)
+int drm_gem_evict_locked(struct drm_gem_object *obj)
{
dma_resv_assert_held(obj->resv);
@@ -1503,4 +1503,4 @@ int drm_gem_evict(struct drm_gem_object *obj)
return 0;
}
-EXPORT_SYMBOL(drm_gem_evict);
+EXPORT_SYMBOL(drm_gem_evict_locked);
diff --git a/include/drm/drm_gem.h b/include/drm/drm_gem.h
index f338f8cfacf7..e78e6d817451 100644
--- a/include/drm/drm_gem.h
+++ b/include/drm/drm_gem.h
@@ -542,7 +542,7 @@ unsigned long drm_gem_lru_scan(struct drm_gem_lru *lru,
unsigned long *remaining,
bool (*shrink)(struct drm_gem_object *obj));
-int drm_gem_evict(struct drm_gem_object *obj);
+int drm_gem_evict_locked(struct drm_gem_object *obj);
#ifdef CONFIG_LOCKDEP
/**
--
2.41.0
[Intel-gfx] [PATCH v15 06/23] drm/virtio: Replace drm_gem_shmem_free() with drm_gem_object_put()
Prepare virtio_gpu_object_create() to addition of memory shrinker support by replacing open-coded drm_gem_shmem_free() with drm_gem_object_put() that decrements GEM refcount to 0, which becomes important for drm-shmem because it will start to use GEM's refcount during the shmem's BO freeing time in order to prevent spurious lockdep warning about resv lock ordering vs fs_reclaim code paths. Signed-off-by: Dmitry Osipenko --- drivers/gpu/drm/virtio/virtgpu_object.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/gpu/drm/virtio/virtgpu_object.c b/drivers/gpu/drm/virtio/virtgpu_object.c index c7e74cf13022..343b13428125 100644 --- a/drivers/gpu/drm/virtio/virtgpu_object.c +++ b/drivers/gpu/drm/virtio/virtgpu_object.c @@ -244,6 +244,6 @@ int virtio_gpu_object_create(struct virtio_gpu_device *vgdev, err_put_id: virtio_gpu_resource_id_put(vgdev, bo->hw_res_handle); err_free_gem: - drm_gem_shmem_free(shmem_obj); + drm_gem_object_put(>base.base); return ret; } -- 2.41.0
[Intel-gfx] [PATCH v15 13/23] drm/shmem-helper: Use kref for pages_use_count
Use atomic kref helper for pages_use_count to optimize pin/unpin functions
by skipping reservation locking while GEM's pin refcount > 1.
Suggested-by: Boris Brezillon
Signed-off-by: Dmitry Osipenko
---
drivers/gpu/drm/drm_gem_shmem_helper.c | 48 ++---
drivers/gpu/drm/lima/lima_gem.c | 2 +-
drivers/gpu/drm/panfrost/panfrost_mmu.c | 2 +-
include/drm/drm_gem_shmem_helper.h | 2 +-
4 files changed, 30 insertions(+), 24 deletions(-)
diff --git a/drivers/gpu/drm/drm_gem_shmem_helper.c
b/drivers/gpu/drm/drm_gem_shmem_helper.c
index 1a7e5c332fd8..5a2e37b3e51d 100644
--- a/drivers/gpu/drm/drm_gem_shmem_helper.c
+++ b/drivers/gpu/drm/drm_gem_shmem_helper.c
@@ -155,7 +155,7 @@ void drm_gem_shmem_free(struct drm_gem_shmem_object *shmem)
if (shmem->got_sgt)
drm_gem_shmem_put_pages_locked(shmem);
- drm_WARN_ON(obj->dev, shmem->pages_use_count);
+ drm_WARN_ON(obj->dev, kref_read(>pages_use_count));
dma_resv_unlock(shmem->base.resv);
}
@@ -172,14 +172,13 @@ static int drm_gem_shmem_get_pages_locked(struct
drm_gem_shmem_object *shmem)
dma_resv_assert_held(shmem->base.resv);
- if (shmem->pages_use_count++ > 0)
+ if (kref_get_unless_zero(>pages_use_count))
return 0;
pages = drm_gem_get_pages(obj);
if (IS_ERR(pages)) {
drm_dbg_kms(obj->dev, "Failed to get pages (%ld)\n",
PTR_ERR(pages));
- shmem->pages_use_count = 0;
return PTR_ERR(pages);
}
@@ -195,26 +194,20 @@ static int drm_gem_shmem_get_pages_locked(struct
drm_gem_shmem_object *shmem)
shmem->pages = pages;
+ kref_init(>pages_use_count);
+
return 0;
}
-/*
- * drm_gem_shmem_put_pages_locked - Decrease use count on the backing pages
for a shmem GEM object
- * @shmem: shmem GEM object
- *
- * This function decreases the use count and puts the backing pages when use
drops to zero.
- */
-void drm_gem_shmem_put_pages_locked(struct drm_gem_shmem_object *shmem)
-{
- struct drm_gem_object *obj = >base;
-
- dma_resv_assert_held(shmem->base.resv);
- if (drm_WARN_ON_ONCE(obj->dev, !shmem->pages_use_count))
- return;
+static void drm_gem_shmem_kref_release_pages(struct kref *kref)
+{
+ struct drm_gem_shmem_object *shmem;
+ struct drm_gem_object *obj;
- if (--shmem->pages_use_count > 0)
- return;
+ shmem = container_of(kref, struct drm_gem_shmem_object,
+pages_use_count);
+ obj = >base;
#ifdef CONFIG_X86
if (shmem->map_wc)
@@ -226,6 +219,19 @@ void drm_gem_shmem_put_pages_locked(struct
drm_gem_shmem_object *shmem)
shmem->pages_mark_accessed_on_put);
shmem->pages = NULL;
}
+
+/*
+ * drm_gem_shmem_put_pages_locked - Decrease use count on the backing pages
for a shmem GEM object
+ * @shmem: shmem GEM object
+ *
+ * This function decreases the use count and puts the backing pages when use
drops to zero.
+ */
+void drm_gem_shmem_put_pages_locked(struct drm_gem_shmem_object *shmem)
+{
+ dma_resv_assert_held(shmem->base.resv);
+
+ kref_put(>pages_use_count, drm_gem_shmem_kref_release_pages);
+}
EXPORT_SYMBOL_GPL(drm_gem_shmem_put_pages_locked);
static int drm_gem_shmem_pin_locked(struct drm_gem_shmem_object *shmem)
@@ -556,8 +562,8 @@ static void drm_gem_shmem_vm_open(struct vm_area_struct
*vma)
* mmap'd, vm_open() just grabs an additional reference for the new
* mm the vma is getting copied into (ie. on fork()).
*/
- if (!drm_WARN_ON_ONCE(obj->dev, !shmem->pages_use_count))
- shmem->pages_use_count++;
+ drm_WARN_ON_ONCE(obj->dev,
+!kref_get_unless_zero(>pages_use_count));
dma_resv_unlock(shmem->base.resv);
@@ -638,7 +644,7 @@ void drm_gem_shmem_print_info(const struct
drm_gem_shmem_object *shmem,
if (shmem->base.import_attach)
return;
- drm_printf_indent(p, indent, "pages_use_count=%u\n",
shmem->pages_use_count);
+ drm_printf_indent(p, indent, "pages_use_count=%u\n",
kref_read(>pages_use_count));
drm_printf_indent(p, indent, "vmap_use_count=%u\n",
shmem->vmap_use_count);
drm_printf_indent(p, indent, "vaddr=%p\n", shmem->vaddr);
}
diff --git a/drivers/gpu/drm/lima/lima_gem.c b/drivers/gpu/drm/lima/lima_gem.c
index 7d74c71f5558..a5f015d188cd 100644
--- a/drivers/gpu/drm/lima/lima_gem.c
+++ b/drivers/gpu/drm/lima/lima_gem.c
@@ -47,7 +47,7 @@ int lima_heap_alloc(struct lima_bo *bo, struct lima_vm *vm)
}
bo->base.pages = pages;
- bo->base.pages_use_count
[Intel-gfx] [PATCH v15 11/23] dma-resv: Add kref_put_dma_resv()
Add simple kref_put_dma_resv() helper that wraps around kref_put_ww_mutex()
for drivers that needs to lock dma-resv on kref_put().
It's not possible to easily add this helper to kref.h because of the
headers inclusion dependency, hence add it to dma-resv.h.
Signed-off-by: Dmitry Osipenko
---
include/linux/dma-resv.h | 9 +
1 file changed, 9 insertions(+)
diff --git a/include/linux/dma-resv.h b/include/linux/dma-resv.h
index 8d0e34dad446..c5cf302e4194 100644
--- a/include/linux/dma-resv.h
+++ b/include/linux/dma-resv.h
@@ -41,6 +41,7 @@
#include
#include
+#include
#include
#include
#include
@@ -464,6 +465,14 @@ static inline void dma_resv_unlock(struct dma_resv *obj)
ww_mutex_unlock(>lock);
}
+static inline int kref_put_dma_resv(struct kref *kref,
+ void (*release)(struct kref *kref),
+ struct dma_resv *resv,
+ struct ww_acquire_ctx *ctx)
+{
+ return kref_put_ww_mutex(kref, release, >lock, ctx);
+}
+
void dma_resv_init(struct dma_resv *obj);
void dma_resv_fini(struct dma_resv *obj);
int dma_resv_reserve_fences(struct dma_resv *obj, unsigned int num_fences);
--
2.41.0
[Intel-gfx] [PATCH v15 05/23] drm/v3d: Replace open-coded drm_gem_shmem_free() with drm_gem_object_put()
The drm_gem_shmem_free() doesn't put GEM's kref to zero, which becomes
important with addition of the shrinker support to drm-shmem that will
use kref=0 in order to prevent taking lock during special GEM-freeing
time in order to avoid spurious lockdep warning about locking ordering
vs fs_reclaim code paths.
Replace open-coded drm_gem_shmem_free() with drm_gem_object_put() that
drops kref to zero before freeing GEM.
Signed-off-by: Dmitry Osipenko
---
drivers/gpu/drm/v3d/v3d_bo.c | 22 --
1 file changed, 12 insertions(+), 10 deletions(-)
diff --git a/drivers/gpu/drm/v3d/v3d_bo.c b/drivers/gpu/drm/v3d/v3d_bo.c
index 8b3229a37c6d..70c1095d6eec 100644
--- a/drivers/gpu/drm/v3d/v3d_bo.c
+++ b/drivers/gpu/drm/v3d/v3d_bo.c
@@ -33,16 +33,18 @@ void v3d_free_object(struct drm_gem_object *obj)
struct v3d_dev *v3d = to_v3d_dev(obj->dev);
struct v3d_bo *bo = to_v3d_bo(obj);
- v3d_mmu_remove_ptes(bo);
+ if (drm_mm_node_allocated(>node)) {
+ v3d_mmu_remove_ptes(bo);
- mutex_lock(>bo_lock);
- v3d->bo_stats.num_allocated--;
- v3d->bo_stats.pages_allocated -= obj->size >> PAGE_SHIFT;
- mutex_unlock(>bo_lock);
+ mutex_lock(>bo_lock);
+ v3d->bo_stats.num_allocated--;
+ v3d->bo_stats.pages_allocated -= obj->size >> PAGE_SHIFT;
+ mutex_unlock(>bo_lock);
- spin_lock(>mm_lock);
- drm_mm_remove_node(>node);
- spin_unlock(>mm_lock);
+ spin_lock(>mm_lock);
+ drm_mm_remove_node(>node);
+ spin_unlock(>mm_lock);
+ }
/* GPU execution may have dirtied any pages in the BO. */
bo->base.pages_mark_dirty_on_put = true;
@@ -142,7 +144,7 @@ struct v3d_bo *v3d_bo_create(struct drm_device *dev, struct
drm_file *file_priv,
return bo;
free_obj:
- drm_gem_shmem_free(shmem_obj);
+ drm_gem_object_put(_obj->base);
return ERR_PTR(ret);
}
@@ -160,7 +162,7 @@ v3d_prime_import_sg_table(struct drm_device *dev,
ret = v3d_bo_create_finish(obj);
if (ret) {
- drm_gem_shmem_free(_v3d_bo(obj)->base);
+ drm_gem_object_put(obj);
return ERR_PTR(ret);
}
--
2.41.0
[Intel-gfx] [PATCH v15 10/23] locking/refcount, kref: Add kref_put_ww_mutex()
Introduce kref_put_ww_mutex() helper that will handle the wait-wound
mutex auto-locking on kref_put(). This helper is wanted by DRM drivers
that extensively use dma-reservation locking which in turns uses ww-mutex.
Signed-off-by: Dmitry Osipenko
---
include/linux/kref.h | 12
include/linux/refcount.h | 5 +
lib/refcount.c | 34 ++
3 files changed, 51 insertions(+)
diff --git a/include/linux/kref.h b/include/linux/kref.h
index d32e21a2538c..b2d8dc6e9ae0 100644
--- a/include/linux/kref.h
+++ b/include/linux/kref.h
@@ -90,6 +90,18 @@ static inline int kref_put_lock(struct kref *kref,
return 0;
}
+static inline int kref_put_ww_mutex(struct kref *kref,
+ void (*release)(struct kref *kref),
+ struct ww_mutex *lock,
+ struct ww_acquire_ctx *ctx)
+{
+ if (refcount_dec_and_ww_mutex_lock(>refcount, lock, ctx)) {
+ release(kref);
+ return 1;
+ }
+ return 0;
+}
+
/**
* kref_get_unless_zero - Increment refcount for object unless it is zero.
* @kref: object.
diff --git a/include/linux/refcount.h b/include/linux/refcount.h
index a62fcca97486..be9ad272bc77 100644
--- a/include/linux/refcount.h
+++ b/include/linux/refcount.h
@@ -99,6 +99,8 @@
#include
struct mutex;
+struct ww_mutex;
+struct ww_acquire_ctx;
/**
* typedef refcount_t - variant of atomic_t specialized for reference counts
@@ -366,4 +368,7 @@ extern __must_check bool refcount_dec_and_lock(refcount_t
*r, spinlock_t *lock)
extern __must_check bool refcount_dec_and_lock_irqsave(refcount_t *r,
spinlock_t *lock,
unsigned long *flags)
__cond_acquires(lock);
+extern __must_check bool refcount_dec_and_ww_mutex_lock(refcount_t *r,
+ struct ww_mutex *lock,
+ struct ww_acquire_ctx
*ctx) __cond_acquires(>base);
#endif /* _LINUX_REFCOUNT_H */
diff --git a/lib/refcount.c b/lib/refcount.c
index a207a8f22b3c..3f6fd0ceed02 100644
--- a/lib/refcount.c
+++ b/lib/refcount.c
@@ -6,6 +6,7 @@
#include
#include
#include
+#include
#include
#define REFCOUNT_WARN(str) WARN_ONCE(1, "refcount_t: " str ".\n")
@@ -184,3 +185,36 @@ bool refcount_dec_and_lock_irqsave(refcount_t *r,
spinlock_t *lock,
return true;
}
EXPORT_SYMBOL(refcount_dec_and_lock_irqsave);
+
+/**
+ * refcount_dec_and_ww_mutex_lock - return holding ww-mutex if able to
+ * decrement refcount to 0
+ * @r: the refcount
+ * @lock: the ww-mutex to be locked
+ * @ctx: wait-wound context
+ *
+ * Similar to atomic_dec_and_lock(), it will WARN on underflow and fail to
+ * decrement when saturated at REFCOUNT_SATURATED.
+ *
+ * Provides release memory ordering, such that prior loads and stores are done
+ * before, and provides a control dependency such that free() must come after.
+ * See the comment on top.
+ *
+ * Return: true and hold ww-mutex lock if able to decrement refcount to 0,
+ * false otherwise
+ */
+bool refcount_dec_and_ww_mutex_lock(refcount_t *r, struct ww_mutex *lock,
+ struct ww_acquire_ctx *ctx)
+{
+ if (refcount_dec_not_one(r))
+ return false;
+
+ ww_mutex_lock(lock, ctx);
+ if (!refcount_dec_and_test(r)) {
+ ww_mutex_unlock(lock);
+ return false;
+ }
+
+ return true;
+}
+EXPORT_SYMBOL(refcount_dec_and_ww_mutex_lock);
--
2.41.0
[Intel-gfx] [PATCH v15 07/23] drm/shmem-helper: Make all exported symbols GPL
Make all drm-shmem exported symbols GPL to make them consistent with
the rest of drm-shmem symbols.
Signed-off-by: Dmitry Osipenko
---
drivers/gpu/drm/drm_gem_shmem_helper.c | 16
1 file changed, 8 insertions(+), 8 deletions(-)
diff --git a/drivers/gpu/drm/drm_gem_shmem_helper.c
b/drivers/gpu/drm/drm_gem_shmem_helper.c
index db20b9123891..575704f38808 100644
--- a/drivers/gpu/drm/drm_gem_shmem_helper.c
+++ b/drivers/gpu/drm/drm_gem_shmem_helper.c
@@ -226,7 +226,7 @@ void drm_gem_shmem_put_pages(struct drm_gem_shmem_object
*shmem)
shmem->pages_mark_accessed_on_put);
shmem->pages = NULL;
}
-EXPORT_SYMBOL(drm_gem_shmem_put_pages);
+EXPORT_SYMBOL_GPL(drm_gem_shmem_put_pages);
static int drm_gem_shmem_pin_locked(struct drm_gem_shmem_object *shmem)
{
@@ -271,7 +271,7 @@ int drm_gem_shmem_pin(struct drm_gem_shmem_object *shmem)
return ret;
}
-EXPORT_SYMBOL(drm_gem_shmem_pin);
+EXPORT_SYMBOL_GPL(drm_gem_shmem_pin);
/**
* drm_gem_shmem_unpin - Unpin backing pages for a shmem GEM object
@@ -290,7 +290,7 @@ void drm_gem_shmem_unpin(struct drm_gem_shmem_object *shmem)
drm_gem_shmem_unpin_locked(shmem);
dma_resv_unlock(shmem->base.resv);
}
-EXPORT_SYMBOL(drm_gem_shmem_unpin);
+EXPORT_SYMBOL_GPL(drm_gem_shmem_unpin);
/*
* drm_gem_shmem_vmap - Create a virtual mapping for a shmem GEM object
@@ -360,7 +360,7 @@ int drm_gem_shmem_vmap(struct drm_gem_shmem_object *shmem,
return ret;
}
-EXPORT_SYMBOL(drm_gem_shmem_vmap);
+EXPORT_SYMBOL_GPL(drm_gem_shmem_vmap);
/*
* drm_gem_shmem_vunmap - Unmap a virtual mapping for a shmem GEM object
@@ -396,7 +396,7 @@ void drm_gem_shmem_vunmap(struct drm_gem_shmem_object
*shmem,
shmem->vaddr = NULL;
}
-EXPORT_SYMBOL(drm_gem_shmem_vunmap);
+EXPORT_SYMBOL_GPL(drm_gem_shmem_vunmap);
static int
drm_gem_shmem_create_with_handle(struct drm_file *file_priv,
@@ -435,7 +435,7 @@ int drm_gem_shmem_madvise(struct drm_gem_shmem_object
*shmem, int madv)
return (madv >= 0);
}
-EXPORT_SYMBOL(drm_gem_shmem_madvise);
+EXPORT_SYMBOL_GPL(drm_gem_shmem_madvise);
void drm_gem_shmem_purge(struct drm_gem_shmem_object *shmem)
{
@@ -467,7 +467,7 @@ void drm_gem_shmem_purge(struct drm_gem_shmem_object *shmem)
invalidate_mapping_pages(file_inode(obj->filp)->i_mapping, 0,
(loff_t)-1);
}
-EXPORT_SYMBOL(drm_gem_shmem_purge);
+EXPORT_SYMBOL_GPL(drm_gem_shmem_purge);
/**
* drm_gem_shmem_dumb_create - Create a dumb shmem buffer object
@@ -636,7 +636,7 @@ void drm_gem_shmem_print_info(const struct
drm_gem_shmem_object *shmem,
drm_printf_indent(p, indent, "vmap_use_count=%u\n",
shmem->vmap_use_count);
drm_printf_indent(p, indent, "vaddr=%p\n", shmem->vaddr);
}
-EXPORT_SYMBOL(drm_gem_shmem_print_info);
+EXPORT_SYMBOL_GPL(drm_gem_shmem_print_info);
/**
* drm_gem_shmem_get_sg_table - Provide a scatter/gather table of pinned
--
2.41.0
[Intel-gfx] [PATCH v15 03/23] drm/gem: Change locked/unlocked postfix of drm_gem_v/unmap() function names
Make drm/gem API function names consistent by having locked function
use the _locked postfix in the name, while the unlocked variants don't
use the _unlocked postfix. Rename drm_gem_v/unmap() function names to
make them consistent with the rest of the API functions.
Suggested-by: Boris Brezillon
Signed-off-by: Dmitry Osipenko
---
drivers/gpu/drm/drm_client.c | 6 +++---
drivers/gpu/drm/drm_gem.c| 20 ++--
drivers/gpu/drm/drm_gem_framebuffer_helper.c | 6 +++---
drivers/gpu/drm/drm_internal.h | 4 ++--
drivers/gpu/drm/drm_prime.c | 4 ++--
drivers/gpu/drm/lima/lima_sched.c| 4 ++--
drivers/gpu/drm/panfrost/panfrost_dump.c | 4 ++--
drivers/gpu/drm/panfrost/panfrost_perfcnt.c | 6 +++---
include/drm/drm_gem.h| 4 ++--
9 files changed, 29 insertions(+), 29 deletions(-)
diff --git a/drivers/gpu/drm/drm_client.c b/drivers/gpu/drm/drm_client.c
index 037e36f2049c..29306657117a 100644
--- a/drivers/gpu/drm/drm_client.c
+++ b/drivers/gpu/drm/drm_client.c
@@ -265,7 +265,7 @@ void drm_client_dev_restore(struct drm_device *dev)
static void drm_client_buffer_delete(struct drm_client_buffer *buffer)
{
if (buffer->gem) {
- drm_gem_vunmap_unlocked(buffer->gem, >map);
+ drm_gem_vunmap(buffer->gem, >map);
drm_gem_object_put(buffer->gem);
}
@@ -349,7 +349,7 @@ drm_client_buffer_vmap(struct drm_client_buffer *buffer,
* fd_install step out of the driver backend hooks, to make that
* final step optional for internal users.
*/
- ret = drm_gem_vmap_unlocked(buffer->gem, map);
+ ret = drm_gem_vmap(buffer->gem, map);
if (ret)
return ret;
@@ -371,7 +371,7 @@ void drm_client_buffer_vunmap(struct drm_client_buffer
*buffer)
{
struct iosys_map *map = >map;
- drm_gem_vunmap_unlocked(buffer->gem, map);
+ drm_gem_vunmap(buffer->gem, map);
}
EXPORT_SYMBOL(drm_client_buffer_vunmap);
diff --git a/drivers/gpu/drm/drm_gem.c b/drivers/gpu/drm/drm_gem.c
index 6129b89bb366..fae5832bb0bd 100644
--- a/drivers/gpu/drm/drm_gem.c
+++ b/drivers/gpu/drm/drm_gem.c
@@ -1173,7 +1173,7 @@ void drm_gem_unpin(struct drm_gem_object *obj)
obj->funcs->unpin(obj);
}
-int drm_gem_vmap(struct drm_gem_object *obj, struct iosys_map *map)
+int drm_gem_vmap_locked(struct drm_gem_object *obj, struct iosys_map *map)
{
int ret;
@@ -1190,9 +1190,9 @@ int drm_gem_vmap(struct drm_gem_object *obj, struct
iosys_map *map)
return 0;
}
-EXPORT_SYMBOL(drm_gem_vmap);
+EXPORT_SYMBOL(drm_gem_vmap_locked);
-void drm_gem_vunmap(struct drm_gem_object *obj, struct iosys_map *map)
+void drm_gem_vunmap_locked(struct drm_gem_object *obj, struct iosys_map *map)
{
dma_resv_assert_held(obj->resv);
@@ -1205,27 +1205,27 @@ void drm_gem_vunmap(struct drm_gem_object *obj, struct
iosys_map *map)
/* Always set the mapping to NULL. Callers may rely on this. */
iosys_map_clear(map);
}
-EXPORT_SYMBOL(drm_gem_vunmap);
+EXPORT_SYMBOL(drm_gem_vunmap_locked);
-int drm_gem_vmap_unlocked(struct drm_gem_object *obj, struct iosys_map *map)
+int drm_gem_vmap(struct drm_gem_object *obj, struct iosys_map *map)
{
int ret;
dma_resv_lock(obj->resv, NULL);
- ret = drm_gem_vmap(obj, map);
+ ret = drm_gem_vmap_locked(obj, map);
dma_resv_unlock(obj->resv);
return ret;
}
-EXPORT_SYMBOL(drm_gem_vmap_unlocked);
+EXPORT_SYMBOL(drm_gem_vmap);
-void drm_gem_vunmap_unlocked(struct drm_gem_object *obj, struct iosys_map *map)
+void drm_gem_vunmap(struct drm_gem_object *obj, struct iosys_map *map)
{
dma_resv_lock(obj->resv, NULL);
- drm_gem_vunmap(obj, map);
+ drm_gem_vunmap_locked(obj, map);
dma_resv_unlock(obj->resv);
}
-EXPORT_SYMBOL(drm_gem_vunmap_unlocked);
+EXPORT_SYMBOL(drm_gem_vunmap);
/**
* drm_gem_lock_reservations - Sets up the ww context and acquires
diff --git a/drivers/gpu/drm/drm_gem_framebuffer_helper.c
b/drivers/gpu/drm/drm_gem_framebuffer_helper.c
index 3bdb6ba37ff4..3808f47310bf 100644
--- a/drivers/gpu/drm/drm_gem_framebuffer_helper.c
+++ b/drivers/gpu/drm/drm_gem_framebuffer_helper.c
@@ -362,7 +362,7 @@ int drm_gem_fb_vmap(struct drm_framebuffer *fb, struct
iosys_map *map,
ret = -EINVAL;
goto err_drm_gem_vunmap;
}
- ret = drm_gem_vmap_unlocked(obj, [i]);
+ ret = drm_gem_vmap(obj, [i]);
if (ret)
goto err_drm_gem_vunmap;
}
@@ -384,7 +384,7 @@ int drm_gem_fb_vmap(struct drm_framebuffer *fb, struct
iosys_map *map,
obj = drm_gem_fb_get_obj(fb, i);
if (!obj)
continue;
- drm_gem_vunmap_unlo
[Intel-gfx] [PATCH v15 01/23] drm/shmem-helper: Fix UAF in error path when freeing SGT of imported GEM
Freeing drm-shmem GEM right after creating it using
drm_gem_shmem_prime_import_sg_table() frees SGT of the imported dma-buf
and then dma-buf frees this SGT second time.
The v3d_prime_import_sg_table() is example of a error code path where
dma-buf's SGT is freed by drm-shmem and then it's freed second time by
dma_buf_unmap_attachment() in drm_gem_prime_import_dev().
Add drm-shmem GEM flag telling that this is imported SGT shall not be
treated as own SGT, fixing the use-after-free bug.
Cc: sta...@vger.kernel.org
Fixes: 2194a63a818d ("drm: Add library for shmem backed GEM objects")
Signed-off-by: Dmitry Osipenko
---
drivers/gpu/drm/drm_gem_shmem_helper.c | 3 ++-
include/drm/drm_gem_shmem_helper.h | 7 +++
2 files changed, 9 insertions(+), 1 deletion(-)
diff --git a/drivers/gpu/drm/drm_gem_shmem_helper.c
b/drivers/gpu/drm/drm_gem_shmem_helper.c
index a783d2245599..78d9cf2355a5 100644
--- a/drivers/gpu/drm/drm_gem_shmem_helper.c
+++ b/drivers/gpu/drm/drm_gem_shmem_helper.c
@@ -141,7 +141,7 @@ void drm_gem_shmem_free(struct drm_gem_shmem_object *shmem)
if (obj->import_attach) {
drm_prime_gem_destroy(obj, shmem->sgt);
- } else {
+ } else if (!shmem->imported_sgt) {
dma_resv_lock(shmem->base.resv, NULL);
drm_WARN_ON(obj->dev, shmem->vmap_use_count);
@@ -758,6 +758,7 @@ drm_gem_shmem_prime_import_sg_table(struct drm_device *dev,
return ERR_CAST(shmem);
shmem->sgt = sgt;
+ shmem->imported_sgt = true;
drm_dbg_prime(dev, "size = %zu\n", size);
diff --git a/include/drm/drm_gem_shmem_helper.h
b/include/drm/drm_gem_shmem_helper.h
index bf0c31aa8fbe..ec70a98a8fe1 100644
--- a/include/drm/drm_gem_shmem_helper.h
+++ b/include/drm/drm_gem_shmem_helper.h
@@ -73,6 +73,13 @@ struct drm_gem_shmem_object {
*/
unsigned int vmap_use_count;
+ /**
+* @imported_sgt:
+*
+* True if SG table belongs to imported dma-buf.
+*/
+ bool imported_sgt : 1;
+
/**
* @pages_mark_dirty_on_put:
*
--
2.41.0
[Intel-gfx] [PATCH v15 02/23] drm/shmem-helper: Use flag for tracking page count bumped by get_pages_sgt()
Use separate flag for tracking page count bumped by shmem->sgt to avoid
imbalanced page counter during of drm_gem_shmem_free() time. It's fragile
to assume that populated shmem->pages at a freeing time means that the
count was bumped by drm_gem_shmem_get_pages_sgt(), using a flag removes
the ambiguity.
Signed-off-by: Dmitry Osipenko
---
drivers/gpu/drm/drm_gem_shmem_helper.c | 3 ++-
drivers/gpu/drm/lima/lima_gem.c| 1 +
include/drm/drm_gem_shmem_helper.h | 7 +++
3 files changed, 10 insertions(+), 1 deletion(-)
diff --git a/drivers/gpu/drm/drm_gem_shmem_helper.c
b/drivers/gpu/drm/drm_gem_shmem_helper.c
index 78d9cf2355a5..db20b9123891 100644
--- a/drivers/gpu/drm/drm_gem_shmem_helper.c
+++ b/drivers/gpu/drm/drm_gem_shmem_helper.c
@@ -152,7 +152,7 @@ void drm_gem_shmem_free(struct drm_gem_shmem_object *shmem)
sg_free_table(shmem->sgt);
kfree(shmem->sgt);
}
- if (shmem->pages)
+ if (shmem->got_sgt)
drm_gem_shmem_put_pages(shmem);
drm_WARN_ON(obj->dev, shmem->pages_use_count);
@@ -687,6 +687,7 @@ static struct sg_table
*drm_gem_shmem_get_pages_sgt_locked(struct drm_gem_shmem_
if (ret)
goto err_free_sgt;
+ shmem->got_sgt = true;
shmem->sgt = sgt;
return sgt;
diff --git a/drivers/gpu/drm/lima/lima_gem.c b/drivers/gpu/drm/lima/lima_gem.c
index 4f9736e5f929..28602302c281 100644
--- a/drivers/gpu/drm/lima/lima_gem.c
+++ b/drivers/gpu/drm/lima/lima_gem.c
@@ -89,6 +89,7 @@ int lima_heap_alloc(struct lima_bo *bo, struct lima_vm *vm)
}
*bo->base.sgt = sgt;
+ bo->base.got_sgt = true;
if (vm) {
ret = lima_vm_map_bo(vm, bo, old_size >> PAGE_SHIFT);
diff --git a/include/drm/drm_gem_shmem_helper.h
b/include/drm/drm_gem_shmem_helper.h
index ec70a98a8fe1..f87124629bb5 100644
--- a/include/drm/drm_gem_shmem_helper.h
+++ b/include/drm/drm_gem_shmem_helper.h
@@ -73,6 +73,13 @@ struct drm_gem_shmem_object {
*/
unsigned int vmap_use_count;
+ /**
+* @got_sgt:
+*
+* True if SG table was retrieved using drm_gem_shmem_get_pages_sgt()
+*/
+ bool got_sgt : 1;
+
/**
* @imported_sgt:
*
--
2.41.0
[Intel-gfx] [PATCH v15 00/23] Add generic memory shrinker to VirtIO-GPU and Panfrost DRM drivers
This series: 1. Adds common drm-shmem memory shrinker 2. Enables shrinker for VirtIO-GPU driver 3. Switches Panfrost driver to the common shrinker Changelog: v15:- Moved drm-shmem reference counters to use kref that allows to optimize unlocked functions, like was suggested by Boris Brezillon. - Changed drm/gem/shmem function names to use _locked postfix and dropped the _unlocked, making the naming scheme consistent across DRM code, like was suggested by Boris Brezillon. - Added patch that fixes UAF in drm-shmem for drivers that import dma-buf and then release buffer in the import error code path. - Added patch that makes drm-shmem use new flag for SGT's get_pages() refcounting, preventing unbalanced refcounting when GEM is freed. - Fixed guest blob pinning in virtio-gpu driver that was missed previously in the shrinker patch. - Moved VC4 and virtio-gpu drivers to use drm_gem_put() in GEM-creation error code paths, which is now required by drm-shmem and was missed in a previous patch versions. - Virtio-GPU now attaches shmem pages to host on first use and not when BO is created. In older patch versions there was a potential race condition in the BO creation code path where both get_sgt()+object_attach() should've been made under same resv lock, otherwise pages could be evicted before attachment is invoked. - Virtio-GPU and drm-shmem shrinker patches are split into smaller ones. v14:- All the prerequisite reservation locking patches landed upstream, previously were a part of this series in v13 and older. https://lore.kernel.org/dri-devel/20230529223935.2672495-1-dmitry.osipe...@collabora.com/ - Added patches to improve locked/unlocked function names, like was suggested by Boris Brezillon for v13. - Made all exported drm-shmem symbols GPL, like was previously discussed with Thomas Zimmermann on this series. - Improved virtio-gpu shrinker patch. Now it won't detach purged BO when userspace closes GEM. Crosvm (and not qemu) checks res_id on CMD_CTX_DETACH_RESOURCE and prints noisy error message if ID is invalid, which wasn't noticed before. v13:- Updated virtio-gpu shrinker patch to use drm_gem_shmem_object_pin() directly instead of drm_gem_pin() and dropped patch that exported drm_gem_pin() functions, like was requested by Thomas Zimmermann in v12. v12:- Fixed the "no previous prototype for function" warning reported by kernel build bot for v11. - Fixed the missing reservation lock reported by Intel CI for VGEM driver. Other drivers using drm-shmem were affected similarly to VGEM. The problem was in the dma-buf attachment code path that led to drm-shmem pinning function which assumed the held reservation lock by drm_gem_pin(). In the past that code path was causing trouble for i915 driver and we've changed the locking scheme for the attachment code path in the dma-buf core to let exporters to handle the locking themselves. After a closer investigation, I realized that my assumption about testing of dma-buf export code path using Panfrost driver was incorrect. Now I created additional local test to exrecise the Panfrost export path. I also reproduced the issue reported by the Intel CI for v10. It's all fixed now by making the drm_gem_shmem_pin() to take the resv lock by itself. - Patches are based on top of drm-tip, CC'd intel-gfx CI for testing. v11:- Rebased on a recent linux-next. Added new patch as a result: drm/shmem-helper: Export drm_gem_shmem_get_pages_sgt_locked() It's needed by the virtio-gpu driver to swap-in/unevict shmem object, previously get_pages_sgt() didn't use locking. - Separated the "Add memory shrinker" patch into smaller parts to ease the reviewing, as was requested by Thomas Zimmermann: drm/shmem-helper: Factor out pages alloc/release from drm_gem_shmem_get/put_pages() drm/shmem-helper: Add pages_pin_count field drm/shmem-helper: Switch drm_gem_shmem_vmap/vunmap to use pin/unpin drm/shmem-helper: Factor out unpinning part from drm_gem_shmem_purge() - Addessed the v10 review comments from Thomas Zimmermann: return errno instead of bool, sort code alphabetically, rename function and etc minor changes. - Added new patch to remove the "map->is_iomem" from drm-shmem, as was suggested by Thomas Zimmermann. - Added acks and r-b's that were given to v10. v10:- Was partially applied to misc-fixes/next. https://lore.kernel.org/dri-devel/6c16f303-81df-7ebe-85e9-51bb40a8b...@collabora.com/T/ Dmitry Osipenko (23): drm/shmem-helper: Fix UAF in error path when freeing SGT of imported GEM drm/shmem-helper: Use flag for tracking page count bumped by
Re: [Intel-gfx] [PATCH v13 03/10] drm/shmem-helper: Add pages_pin_count field
On 6/26/23 18:21, Boris Brezillon wrote:
> On Mon, 26 Jun 2023 17:04:57 +0200
> Boris Brezillon wrote:
>
>> Hi Dmitry,
>>
>> Sorry for chiming in only now :-/.
>>
>> On Tue, 14 Mar 2023 05:26:52 +0300
>> Dmitry Osipenko wrote:
>>
>>> And new pages_pin_count field to struct drm_gem_shmem_object that will
>>> determine whether pages are evictable by memory shrinker. The pages will
>>> be evictable only when pages_pin_count=0. This patch prepares code for
>>> addition of the memory shrinker that will utilize the new field.
>>>
>>> Signed-off-by: Dmitry Osipenko
>>> ---
>>> drivers/gpu/drm/drm_gem_shmem_helper.c | 7 +++
>>> include/drm/drm_gem_shmem_helper.h | 9 +
>>> 2 files changed, 16 insertions(+)
>>>
>>> diff --git a/drivers/gpu/drm/drm_gem_shmem_helper.c
>>> b/drivers/gpu/drm/drm_gem_shmem_helper.c
>>> index 4da9c9c39b9a..81d61791f874 100644
>>> --- a/drivers/gpu/drm/drm_gem_shmem_helper.c
>>> +++ b/drivers/gpu/drm/drm_gem_shmem_helper.c
>>> @@ -277,6 +277,8 @@ static int drm_gem_shmem_pin_locked(struct
>>> drm_gem_shmem_object *shmem)
>>> drm_WARN_ON(obj->dev, obj->import_attach);
>>>
>>> ret = drm_gem_shmem_get_pages(shmem);
>>> + if (!ret)
>>> + shmem->pages_pin_count++;
>>>
>>> return ret;
>>> }
>>> @@ -289,7 +291,12 @@ static void drm_gem_shmem_unpin_locked(struct
>>> drm_gem_shmem_object *shmem)
>>>
>>> drm_WARN_ON(obj->dev, obj->import_attach);
>>>
>>> + if (drm_WARN_ON_ONCE(obj->dev, !shmem->pages_pin_count))
>>> + return;
>>> +
>>> drm_gem_shmem_put_pages(shmem);
>>> +
>>> + shmem->pages_pin_count--;
>>> }
>>>
>>> /**
>>> diff --git a/include/drm/drm_gem_shmem_helper.h
>>> b/include/drm/drm_gem_shmem_helper.h
>>> index 20ddcd799df9..7d823c9fc480 100644
>>> --- a/include/drm/drm_gem_shmem_helper.h
>>> +++ b/include/drm/drm_gem_shmem_helper.h
>>> @@ -39,6 +39,15 @@ struct drm_gem_shmem_object {
>>> */
>>> unsigned int pages_use_count;
>>>
>>> + /**
>>> +* @pages_pin_count:
>>> +*
>>> +* Reference count on the pinned pages table.
>>> +* The pages allowed to be evicted by memory shrinker
>>> +* only when the count is zero.
>>> +*/
>>> + unsigned int pages_pin_count;
>>
>> s/pages_pin_count/pin_count/ ?
>>
>> And do we really need both pages_pin_count and pages_use_count. Looks
>> like they both serve the same purpose, with one exception:
>> pages_use_count is also incremented in the get_pages_sgt_locked() path,
>> but you probably don't want it to prevent GEM eviction. Assuming
>> your goal with this pin_count field is to check if a GEM object is
>> evictable, it can be done with something like
>>
>> bool
>> drm_gem_shmem_is_evictable_locked(struct drm_gem_shmem_object *shmem)
>> {
>> dma_resv_assert_held(shmem->base.resv);
>>
>> return shmem->pages_use_count == (shmem->sgt ? 1 : 0);
>> }
>>
>> I mean, I'm not against renaming pages_use_count into pin_count, but,
>> unless I'm missing something, I don't see a good reason to keep both.
>
> My bad, I think I found one place calling drm_gem_shmem_get_pages()
> where we want pin_count and pages_use_count to differ:
> drm_gem_shmem_mmap(). We certainly don't want userspace mappings to
> prevent eviction.
That's correct, thanks for the review :)
--
Best regards,
Dmitry
Re: [Intel-gfx] [PATCH v4 6/6] drm/shmem-helper: Switch to reservation lock
On 6/26/23 12:40, Boris Brezillon wrote:
> I think here is the major problem I have with this patch: you've made
> drm_gem_shmem_{get_pages,pin}() private, which forces me to call
> drm_gem_shmem_pin() in a path where I already acquired the resv lock
> (using the drm_exec infra proposed by Christian). That would
> probably work if you were letting ret == -EALREADY go through, but I'm
> wondering if it wouldn't be preferable to expose
> drm_gem_shmem_pin_locked().
You should be free to expose the necessary functions. They are private
because nobody need them so far and we don't want to export unused
functions.
--
Best regards,
Dmitry
Re: [Intel-gfx] [PATCH v4 6/6] drm/shmem-helper: Switch to reservation lock
On 6/26/23 12:40, Boris Brezillon wrote: > Same problem with this renaming: it's confusing because this function > was previously taking care of the locking, and it's no longer the case. > That's actually true for other public functions your patching, but I > won't go over all of them. > > I know this patch has been under discussion for quite some time, and has > been validated by other devs/maintainers, but I'd like to understand the > reasoning behind these decisions. Not the decision to replace all locks > by dma_resv, which I kinda understand, but the decision to change the > behavior of functions without making the name reflect the new behavior > (_locked prefix), or the fact we now prohibit some functions to > succeed when the dma_resv lock is taken by the driver beforehand (which, > unless I'm mistaken, will happen in the VM_BIND logic, and can happen > in the SUBMIT ioctl too depending on the driver). Adding explicit _locked/unlocked postfix to all function names indeed won't hurt to do. There was no decision made about the function names, the old functions kept the old name where possible to minimize code changes during transition to the resv lock. Improving the names could be the next step. Thanks for the feedback! -- Best regards, Dmitry
Re: [Intel-gfx] [PATCH v4 0/6] Move dma-buf mmap() reservation locking down to exporters
On 5/30/23 01:39, Dmitry Osipenko wrote: > This patchset makes dma-buf exporters responisble for taking care of > the reservation lock. I also included patch that moves drm-shmem to use > reservation lock, to let CI test the whole set. I'm going to take all > the patches via the drm-misc tree, please give an ack. > > Previous policy stated that dma-buf core takes the lock around mmap() > callback. Which meant that both importers and exporters shouldn't touch > the reservation lock in the mmap() code path. This worked well until > Intel-CI found a deadlock problem in a case of self-imported dma-buf [1]. > > The problem happens when userpace mmaps a self-imported dma-buf, i.e. > mmaps the dma-buf FD. DRM core treats self-imported dma-bufs as own GEMs > [2]. There is no way to differentiate a prime GEM from a normal GEM for > drm-shmem in drm_gem_shmem_mmap(), which resulted in a deadlock problem > for drm-shmem mmap() code path once it's switched to use reservation lock. > > It was difficult to fix the drm-shmem problem without adjusting dma-buf > locking policy. In parctice not much changed from importers perspective > because previosly dma-buf was taking the lock in between of importers > and exporters. Now this lock is shifted down to exporters. > > [1] > https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_114671v2/shard-snb5/igt@prime_vgem@s...@rcs0.html > [2] > https://elixir.bootlin.com/linux/v6.3-rc4/source/drivers/gpu/drm/drm_prime.c#L924 Applied to misc-next -- Best regards, Dmitry
Re: [Intel-gfx] [PATCH v4 5/6] dma-buf: Change locking policy for mmap()
On 6/21/23 08:42, Christian König wrote: > Am 20.06.23 um 17:58 schrieb Dmitry Osipenko: >> On 5/31/23 22:58, Dmitry Osipenko wrote: >>> On 5/30/23 01:39, Dmitry Osipenko wrote: >>>> Change locking policy of mmap() callback, making exporters responsible >>>> for handling dma-buf reservation locking. Previous locking policy >>>> stated >>>> that dma-buf is locked for both importers and exporters by the dma-buf >>>> core, which caused a deadlock problem for DRM drivers in a case of >>>> self-imported dma-bufs which required to take the lock from the DRM >>>> exporter side. >>>> >>>> Reviewed-by: Emil Velikov >>>> Signed-off-by: Dmitry Osipenko >>>> --- >>>> drivers/dma-buf/dma-buf.c | 17 +++-- >>>> 1 file changed, 3 insertions(+), 14 deletions(-) >>> Christian, you acked the drm patch of this series sometime ago, perhaps >>> it also implies implicit ack to this patch, but I'd prefer to have the >>> explicit ack. I'll apply this series to drm-misc later this week if >>> you'll approve this dma-buf change. Thanks in advance! >> I'll merge the patches tomorrow. If there are any additional comments, >> then please don't hesitate to post them. > > Sorry for not responding earlier, I have been moving both my office as > well as my household and still catching up. > > I don't have time for an in-deep review, but my ack stands for the whole > series. Thanks! I'll add yours ack and merge patches soon -- Best regards, Dmitry
Re: [Intel-gfx] [PATCH v4 2/6] dma-buf/heaps: Don't assert held reservation lock for dma-buf mmapping
Hi, On 6/21/23 20:21, T.J. Mercier wrote: > On Mon, May 29, 2023 at 3:46 PM Dmitry Osipenko > wrote: >> >> Don't assert held dma-buf reservation lock on memory mapping of exported >> buffer. >> >> We're going to change dma-buf mmap() locking policy such that exporters >> will have to handle the lock. The previous locking policy caused deadlock >> problem for DRM drivers in a case of self-imported dma-bufs once these >> drivers are moved to use reservation lock universally. The problem >> solved by moving the lock down to exporters. This patch prepares dma-buf >> heaps for the locking policy update. >> > Hi Dmitry, > > I see that in patch 6 of this series calls to > dma_resv_lock/dma_resv_unlock have been added to the > drm_gem_shmem_helper functions and some exporters. But I'm curious why > no dma_resv_lock/dma_resv_unlock calls were added to these two dma-buf > heap exporters for mmap? DMA-buf heaps are exporters, drm_gem_shmem_helper is importer. Locking rules are different for importers and exporters. DMA-heaps use own locking, they can be moved to resv lock in the future. DMA-heaps don't protect internal data in theirs mmap() implementations, nothing to protect there. -- Best regards, Dmitry
Re: [Intel-gfx] [PATCH v4 5/6] dma-buf: Change locking policy for mmap()
On 5/31/23 22:58, Dmitry Osipenko wrote: > On 5/30/23 01:39, Dmitry Osipenko wrote: >> Change locking policy of mmap() callback, making exporters responsible >> for handling dma-buf reservation locking. Previous locking policy stated >> that dma-buf is locked for both importers and exporters by the dma-buf >> core, which caused a deadlock problem for DRM drivers in a case of >> self-imported dma-bufs which required to take the lock from the DRM >> exporter side. >> >> Reviewed-by: Emil Velikov >> Signed-off-by: Dmitry Osipenko >> --- >> drivers/dma-buf/dma-buf.c | 17 +++-- >> 1 file changed, 3 insertions(+), 14 deletions(-) > > Christian, you acked the drm patch of this series sometime ago, perhaps > it also implies implicit ack to this patch, but I'd prefer to have the > explicit ack. I'll apply this series to drm-misc later this week if > you'll approve this dma-buf change. Thanks in advance! I'll merge the patches tomorrow. If there are any additional comments, then please don't hesitate to post them. -- Best regards, Dmitry
Re: [Intel-gfx] [PATCH v4 5/6] dma-buf: Change locking policy for mmap()
On 5/30/23 01:39, Dmitry Osipenko wrote: > Change locking policy of mmap() callback, making exporters responsible > for handling dma-buf reservation locking. Previous locking policy stated > that dma-buf is locked for both importers and exporters by the dma-buf > core, which caused a deadlock problem for DRM drivers in a case of > self-imported dma-bufs which required to take the lock from the DRM > exporter side. > > Reviewed-by: Emil Velikov > Signed-off-by: Dmitry Osipenko > --- > drivers/dma-buf/dma-buf.c | 17 +++-- > 1 file changed, 3 insertions(+), 14 deletions(-) Christian, you acked the drm patch of this series sometime ago, perhaps it also implies implicit ack to this patch, but I'd prefer to have the explicit ack. I'll apply this series to drm-misc later this week if you'll approve this dma-buf change. Thanks in advance! -- Best regards, Dmitry
[Intel-gfx] [PATCH v4 6/6] drm/shmem-helper: Switch to reservation lock
Replace all drm-shmem locks with a GEM reservation lock. This makes locks
consistent with dma-buf locking convention where importers are responsible
for holding reservation lock for all operations performed over dma-bufs,
preventing deadlock between dma-buf importers and exporters.
Suggested-by: Daniel Vetter
Acked-by: Thomas Zimmermann
Reviewed-by: Emil Velikov
Signed-off-by: Dmitry Osipenko
---
drivers/gpu/drm/drm_gem_shmem_helper.c| 210 --
drivers/gpu/drm/lima/lima_gem.c | 8 +-
drivers/gpu/drm/panfrost/panfrost_drv.c | 7 +-
.../gpu/drm/panfrost/panfrost_gem_shrinker.c | 6 +-
drivers/gpu/drm/panfrost/panfrost_mmu.c | 19 +-
include/drm/drm_gem_shmem_helper.h| 14 +-
6 files changed, 116 insertions(+), 148 deletions(-)
diff --git a/drivers/gpu/drm/drm_gem_shmem_helper.c
b/drivers/gpu/drm/drm_gem_shmem_helper.c
index 4ea6507a77e5..a783d2245599 100644
--- a/drivers/gpu/drm/drm_gem_shmem_helper.c
+++ b/drivers/gpu/drm/drm_gem_shmem_helper.c
@@ -88,8 +88,6 @@ __drm_gem_shmem_create(struct drm_device *dev, size_t size,
bool private)
if (ret)
goto err_release;
- mutex_init(>pages_lock);
- mutex_init(>vmap_lock);
INIT_LIST_HEAD(>madv_list);
if (!private) {
@@ -141,11 +139,13 @@ void drm_gem_shmem_free(struct drm_gem_shmem_object
*shmem)
{
struct drm_gem_object *obj = >base;
- drm_WARN_ON(obj->dev, shmem->vmap_use_count);
-
if (obj->import_attach) {
drm_prime_gem_destroy(obj, shmem->sgt);
} else {
+ dma_resv_lock(shmem->base.resv, NULL);
+
+ drm_WARN_ON(obj->dev, shmem->vmap_use_count);
+
if (shmem->sgt) {
dma_unmap_sgtable(obj->dev->dev, shmem->sgt,
DMA_BIDIRECTIONAL, 0);
@@ -154,22 +154,24 @@ void drm_gem_shmem_free(struct drm_gem_shmem_object
*shmem)
}
if (shmem->pages)
drm_gem_shmem_put_pages(shmem);
- }
- drm_WARN_ON(obj->dev, shmem->pages_use_count);
+ drm_WARN_ON(obj->dev, shmem->pages_use_count);
+
+ dma_resv_unlock(shmem->base.resv);
+ }
drm_gem_object_release(obj);
- mutex_destroy(>pages_lock);
- mutex_destroy(>vmap_lock);
kfree(shmem);
}
EXPORT_SYMBOL_GPL(drm_gem_shmem_free);
-static int drm_gem_shmem_get_pages_locked(struct drm_gem_shmem_object *shmem)
+static int drm_gem_shmem_get_pages(struct drm_gem_shmem_object *shmem)
{
struct drm_gem_object *obj = >base;
struct page **pages;
+ dma_resv_assert_held(shmem->base.resv);
+
if (shmem->pages_use_count++ > 0)
return 0;
@@ -197,35 +199,16 @@ static int drm_gem_shmem_get_pages_locked(struct
drm_gem_shmem_object *shmem)
}
/*
- * drm_gem_shmem_get_pages - Allocate backing pages for a shmem GEM object
+ * drm_gem_shmem_put_pages - Decrease use count on the backing pages for a
shmem GEM object
* @shmem: shmem GEM object
*
- * This function makes sure that backing pages exists for the shmem GEM object
- * and increases the use count.
- *
- * Returns:
- * 0 on success or a negative error code on failure.
+ * This function decreases the use count and puts the backing pages when use
drops to zero.
*/
-int drm_gem_shmem_get_pages(struct drm_gem_shmem_object *shmem)
+void drm_gem_shmem_put_pages(struct drm_gem_shmem_object *shmem)
{
struct drm_gem_object *obj = >base;
- int ret;
- drm_WARN_ON(obj->dev, obj->import_attach);
-
- ret = mutex_lock_interruptible(>pages_lock);
- if (ret)
- return ret;
- ret = drm_gem_shmem_get_pages_locked(shmem);
- mutex_unlock(>pages_lock);
-
- return ret;
-}
-EXPORT_SYMBOL(drm_gem_shmem_get_pages);
-
-static void drm_gem_shmem_put_pages_locked(struct drm_gem_shmem_object *shmem)
-{
- struct drm_gem_object *obj = >base;
+ dma_resv_assert_held(shmem->base.resv);
if (drm_WARN_ON_ONCE(obj->dev, !shmem->pages_use_count))
return;
@@ -243,20 +226,25 @@ static void drm_gem_shmem_put_pages_locked(struct
drm_gem_shmem_object *shmem)
shmem->pages_mark_accessed_on_put);
shmem->pages = NULL;
}
+EXPORT_SYMBOL(drm_gem_shmem_put_pages);
-/*
- * drm_gem_shmem_put_pages - Decrease use count on the backing pages for a
shmem GEM object
- * @shmem: shmem GEM object
- *
- * This function decreases the use count and puts the backing pages when use
drops to zero.
- */
-void drm_gem_shmem_put_pages(struct drm_gem_shmem_object *shmem)
+static int drm_gem_shmem_pin_locked(struct drm_gem_shmem_object *shmem)
{
- mutex_lock(>pages_lock);
- drm_gem_shmem_put_pages_locked(shmem)
[Intel-gfx] [PATCH v4 2/6] dma-buf/heaps: Don't assert held reservation lock for dma-buf mmapping
Don't assert held dma-buf reservation lock on memory mapping of exported
buffer.
We're going to change dma-buf mmap() locking policy such that exporters
will have to handle the lock. The previous locking policy caused deadlock
problem for DRM drivers in a case of self-imported dma-bufs once these
drivers are moved to use reservation lock universally. The problem
solved by moving the lock down to exporters. This patch prepares dma-buf
heaps for the locking policy update.
Reviewed-by: Emil Velikov
Signed-off-by: Dmitry Osipenko
---
drivers/dma-buf/heaps/cma_heap.c| 3 ---
drivers/dma-buf/heaps/system_heap.c | 3 ---
2 files changed, 6 deletions(-)
diff --git a/drivers/dma-buf/heaps/cma_heap.c b/drivers/dma-buf/heaps/cma_heap.c
index 1131fb943992..28fb04eccdd0 100644
--- a/drivers/dma-buf/heaps/cma_heap.c
+++ b/drivers/dma-buf/heaps/cma_heap.c
@@ -13,7 +13,6 @@
#include
#include
#include
-#include
#include
#include
#include
@@ -183,8 +182,6 @@ static int cma_heap_mmap(struct dma_buf *dmabuf, struct
vm_area_struct *vma)
{
struct cma_heap_buffer *buffer = dmabuf->priv;
- dma_resv_assert_held(dmabuf->resv);
-
if ((vma->vm_flags & (VM_SHARED | VM_MAYSHARE)) == 0)
return -EINVAL;
diff --git a/drivers/dma-buf/heaps/system_heap.c
b/drivers/dma-buf/heaps/system_heap.c
index e8bd10e60998..fcf836ba9c1f 100644
--- a/drivers/dma-buf/heaps/system_heap.c
+++ b/drivers/dma-buf/heaps/system_heap.c
@@ -13,7 +13,6 @@
#include
#include
#include
-#include
#include
#include
#include
@@ -202,8 +201,6 @@ static int system_heap_mmap(struct dma_buf *dmabuf, struct
vm_area_struct *vma)
struct sg_page_iter piter;
int ret;
- dma_resv_assert_held(dmabuf->resv);
-
for_each_sgtable_page(table, , vma->vm_pgoff) {
struct page *page = sg_page_iter_page();
--
2.40.1
[Intel-gfx] [PATCH v4 5/6] dma-buf: Change locking policy for mmap()
Change locking policy of mmap() callback, making exporters responsible
for handling dma-buf reservation locking. Previous locking policy stated
that dma-buf is locked for both importers and exporters by the dma-buf
core, which caused a deadlock problem for DRM drivers in a case of
self-imported dma-bufs which required to take the lock from the DRM
exporter side.
Reviewed-by: Emil Velikov
Signed-off-by: Dmitry Osipenko
---
drivers/dma-buf/dma-buf.c | 17 +++--
1 file changed, 3 insertions(+), 14 deletions(-)
diff --git a/drivers/dma-buf/dma-buf.c b/drivers/dma-buf/dma-buf.c
index aa4ea8530cb3..21916bba77d5 100644
--- a/drivers/dma-buf/dma-buf.c
+++ b/drivers/dma-buf/dma-buf.c
@@ -131,7 +131,6 @@ static struct file_system_type dma_buf_fs_type = {
static int dma_buf_mmap_internal(struct file *file, struct vm_area_struct *vma)
{
struct dma_buf *dmabuf;
- int ret;
if (!is_dma_buf_file(file))
return -EINVAL;
@@ -147,11 +146,7 @@ static int dma_buf_mmap_internal(struct file *file, struct
vm_area_struct *vma)
dmabuf->size >> PAGE_SHIFT)
return -EINVAL;
- dma_resv_lock(dmabuf->resv, NULL);
- ret = dmabuf->ops->mmap(dmabuf, vma);
- dma_resv_unlock(dmabuf->resv);
-
- return ret;
+ return dmabuf->ops->mmap(dmabuf, vma);
}
static loff_t dma_buf_llseek(struct file *file, loff_t offset, int whence)
@@ -850,6 +845,7 @@ static struct sg_table * __map_dma_buf(struct
dma_buf_attachment *attach,
* - _buf_ops.release()
* - _buf_ops.begin_cpu_access()
* - _buf_ops.end_cpu_access()
+ * - _buf_ops.mmap()
*
* 2. These _buf_ops callbacks are invoked with locked dma-buf
*reservation and exporter can't take the lock:
@@ -858,7 +854,6 @@ static struct sg_table * __map_dma_buf(struct
dma_buf_attachment *attach,
* - _buf_ops.unpin()
* - _buf_ops.map_dma_buf()
* - _buf_ops.unmap_dma_buf()
- * - _buf_ops.mmap()
* - _buf_ops.vmap()
* - _buf_ops.vunmap()
*
@@ -1463,8 +1458,6 @@ EXPORT_SYMBOL_NS_GPL(dma_buf_end_cpu_access, DMA_BUF);
int dma_buf_mmap(struct dma_buf *dmabuf, struct vm_area_struct *vma,
unsigned long pgoff)
{
- int ret;
-
if (WARN_ON(!dmabuf || !vma))
return -EINVAL;
@@ -1485,11 +1478,7 @@ int dma_buf_mmap(struct dma_buf *dmabuf, struct
vm_area_struct *vma,
vma_set_file(vma, dmabuf->file);
vma->vm_pgoff = pgoff;
- dma_resv_lock(dmabuf->resv, NULL);
- ret = dmabuf->ops->mmap(dmabuf, vma);
- dma_resv_unlock(dmabuf->resv);
-
- return ret;
+ return dmabuf->ops->mmap(dmabuf, vma);
}
EXPORT_SYMBOL_NS_GPL(dma_buf_mmap, DMA_BUF);
--
2.40.1
[Intel-gfx] [PATCH v4 3/6] udmabuf: Don't assert held reservation lock for dma-buf mmapping
Don't assert held dma-buf reservation lock on memory mapping of exported
buffer.
We're going to change dma-buf mmap() locking policy such that exporters
will have to handle the lock. The previous locking policy caused deadlock
problem for DRM drivers in a case of self-imported dma-bufs once these
drivers are moved to use reservation lock universally. The problem is
solved by moving the lock down to exporters. This patch prepares udmabuf
for the locking policy update.
Reviewed-by: Emil Velikov
Signed-off-by: Dmitry Osipenko
---
drivers/dma-buf/udmabuf.c | 2 --
1 file changed, 2 deletions(-)
diff --git a/drivers/dma-buf/udmabuf.c b/drivers/dma-buf/udmabuf.c
index 740d6e426ee9..277f1afa9552 100644
--- a/drivers/dma-buf/udmabuf.c
+++ b/drivers/dma-buf/udmabuf.c
@@ -52,8 +52,6 @@ static int mmap_udmabuf(struct dma_buf *buf, struct
vm_area_struct *vma)
{
struct udmabuf *ubuf = buf->priv;
- dma_resv_assert_held(buf->resv);
-
if ((vma->vm_flags & (VM_SHARED | VM_MAYSHARE)) == 0)
return -EINVAL;
--
2.40.1
[Intel-gfx] [PATCH v4 4/6] drm: Don't assert held reservation lock for dma-buf mmapping
Don't assert held dma-buf reservation lock on memory mapping of exported buffer. We're going to change dma-buf mmap() locking policy such that exporters will have to handle the lock. The previous locking policy caused deadlock problem for DRM drivers in a case of self-imported dma-bufs once these drivers are moved to use reservation lock universally. The problem is solved by moving the lock down to exporters. This patch prepares DRM drivers for the locking policy update. Reviewed-by: Emil Velikov Acked-by: Christian König Signed-off-by: Dmitry Osipenko --- drivers/gpu/drm/drm_prime.c| 2 -- drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c | 2 -- drivers/gpu/drm/omapdrm/omap_gem_dmabuf.c | 2 -- drivers/gpu/drm/tegra/gem.c| 2 -- 4 files changed, 8 deletions(-) diff --git a/drivers/gpu/drm/drm_prime.c b/drivers/gpu/drm/drm_prime.c index 149cd4ff6a3b..cea85e84666f 100644 --- a/drivers/gpu/drm/drm_prime.c +++ b/drivers/gpu/drm/drm_prime.c @@ -781,8 +781,6 @@ int drm_gem_dmabuf_mmap(struct dma_buf *dma_buf, struct vm_area_struct *vma) struct drm_gem_object *obj = dma_buf->priv; struct drm_device *dev = obj->dev; - dma_resv_assert_held(dma_buf->resv); - if (!dev->driver->gem_prime_mmap) return -ENOSYS; diff --git a/drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c b/drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c index fd556a076d05..1df74f7aa3dc 100644 --- a/drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c +++ b/drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c @@ -97,8 +97,6 @@ static int i915_gem_dmabuf_mmap(struct dma_buf *dma_buf, struct vm_area_struct * struct drm_i915_private *i915 = to_i915(obj->base.dev); int ret; - dma_resv_assert_held(dma_buf->resv); - if (obj->base.size < vma->vm_end - vma->vm_start) return -EINVAL; diff --git a/drivers/gpu/drm/omapdrm/omap_gem_dmabuf.c b/drivers/gpu/drm/omapdrm/omap_gem_dmabuf.c index 3abc47521b2c..8e194dbc9506 100644 --- a/drivers/gpu/drm/omapdrm/omap_gem_dmabuf.c +++ b/drivers/gpu/drm/omapdrm/omap_gem_dmabuf.c @@ -66,8 +66,6 @@ static int omap_gem_dmabuf_mmap(struct dma_buf *buffer, struct drm_gem_object *obj = buffer->priv; int ret = 0; - dma_resv_assert_held(buffer->resv); - ret = drm_gem_mmap_obj(obj, omap_gem_mmap_size(obj), vma); if (ret < 0) return ret; diff --git a/drivers/gpu/drm/tegra/gem.c b/drivers/gpu/drm/tegra/gem.c index bce991a2ccc0..871ef5d26523 100644 --- a/drivers/gpu/drm/tegra/gem.c +++ b/drivers/gpu/drm/tegra/gem.c @@ -693,8 +693,6 @@ static int tegra_gem_prime_mmap(struct dma_buf *buf, struct vm_area_struct *vma) struct drm_gem_object *gem = buf->priv; int err; - dma_resv_assert_held(buf->resv); - err = drm_gem_mmap_obj(gem, gem->size, vma); if (err < 0) return err; -- 2.40.1
[Intel-gfx] [PATCH v4 1/6] media: videobuf2: Don't assert held reservation lock for dma-buf mmapping
Don't assert held dma-buf reservation lock on memory mapping of exported
buffer.
We're going to change dma-buf mmap() locking policy such that exporters
will have to handle the lock. The previous locking policy caused deadlock
problem for DRM drivers in a case of self-imported dma-bufs once these
drivers are moved to use reservation lock universally. The problem is
solved by moving the lock down to exporters. This patch prepares videobuf2
for the locking policy update.
Reviewed-by: Emil Velikov
Reviewed-by: Hans Verkuil
Signed-off-by: Dmitry Osipenko
---
drivers/media/common/videobuf2/videobuf2-dma-contig.c | 3 ---
drivers/media/common/videobuf2/videobuf2-dma-sg.c | 3 ---
drivers/media/common/videobuf2/videobuf2-vmalloc.c| 3 ---
3 files changed, 9 deletions(-)
diff --git a/drivers/media/common/videobuf2/videobuf2-dma-contig.c
b/drivers/media/common/videobuf2/videobuf2-dma-contig.c
index 205d3cac425c..2fa455d4a048 100644
--- a/drivers/media/common/videobuf2/videobuf2-dma-contig.c
+++ b/drivers/media/common/videobuf2/videobuf2-dma-contig.c
@@ -11,7 +11,6 @@
*/
#include
-#include
#include
#include
#include
@@ -456,8 +455,6 @@ static int vb2_dc_dmabuf_ops_vmap(struct dma_buf *dbuf,
struct iosys_map *map)
static int vb2_dc_dmabuf_ops_mmap(struct dma_buf *dbuf,
struct vm_area_struct *vma)
{
- dma_resv_assert_held(dbuf->resv);
-
return vb2_dc_mmap(dbuf->priv, vma);
}
diff --git a/drivers/media/common/videobuf2/videobuf2-dma-sg.c
b/drivers/media/common/videobuf2/videobuf2-dma-sg.c
index 183037fb1273..28f3fdfe23a2 100644
--- a/drivers/media/common/videobuf2/videobuf2-dma-sg.c
+++ b/drivers/media/common/videobuf2/videobuf2-dma-sg.c
@@ -10,7 +10,6 @@
* the Free Software Foundation.
*/
-#include
#include
#include
#include
@@ -498,8 +497,6 @@ static int vb2_dma_sg_dmabuf_ops_vmap(struct dma_buf *dbuf,
static int vb2_dma_sg_dmabuf_ops_mmap(struct dma_buf *dbuf,
struct vm_area_struct *vma)
{
- dma_resv_assert_held(dbuf->resv);
-
return vb2_dma_sg_mmap(dbuf->priv, vma);
}
diff --git a/drivers/media/common/videobuf2/videobuf2-vmalloc.c
b/drivers/media/common/videobuf2/videobuf2-vmalloc.c
index a6c6d2fcaaa4..7c635e292106 100644
--- a/drivers/media/common/videobuf2/videobuf2-vmalloc.c
+++ b/drivers/media/common/videobuf2/videobuf2-vmalloc.c
@@ -10,7 +10,6 @@
* the Free Software Foundation.
*/
-#include
#include
#include
#include
@@ -319,8 +318,6 @@ static int vb2_vmalloc_dmabuf_ops_vmap(struct dma_buf *dbuf,
static int vb2_vmalloc_dmabuf_ops_mmap(struct dma_buf *dbuf,
struct vm_area_struct *vma)
{
- dma_resv_assert_held(dbuf->resv);
-
return vb2_vmalloc_mmap(dbuf->priv, vma);
}
--
2.40.1
[Intel-gfx] [PATCH v4 0/6] Move dma-buf mmap() reservation locking down to exporters
This patchset makes dma-buf exporters responisble for taking care of the reservation lock. I also included patch that moves drm-shmem to use reservation lock, to let CI test the whole set. I'm going to take all the patches via the drm-misc tree, please give an ack. Previous policy stated that dma-buf core takes the lock around mmap() callback. Which meant that both importers and exporters shouldn't touch the reservation lock in the mmap() code path. This worked well until Intel-CI found a deadlock problem in a case of self-imported dma-buf [1]. The problem happens when userpace mmaps a self-imported dma-buf, i.e. mmaps the dma-buf FD. DRM core treats self-imported dma-bufs as own GEMs [2]. There is no way to differentiate a prime GEM from a normal GEM for drm-shmem in drm_gem_shmem_mmap(), which resulted in a deadlock problem for drm-shmem mmap() code path once it's switched to use reservation lock. It was difficult to fix the drm-shmem problem without adjusting dma-buf locking policy. In parctice not much changed from importers perspective because previosly dma-buf was taking the lock in between of importers and exporters. Now this lock is shifted down to exporters. [1] https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_114671v2/shard-snb5/igt@prime_vgem@s...@rcs0.html [2] https://elixir.bootlin.com/linux/v6.3-rc4/source/drivers/gpu/drm/drm_prime.c#L924 Changelog: v4: - Added dma_resv_assert_held() to drm_gem_shmem_get_pages(), making assert consistent with the put() variant. Suggested by Emil Velikov. v3: - Added r-b from Hans Verkuil to the videobuf2 patch. - The v2 fastrpc patch was already applied, not including it anymore. Though, the cover-letter says that I'd want to apply all the patches via the drm-misc tree to keep the proper ordering of the changes. - Previously Intel's CI gave a flake failure to v2, want to re-test it again. v2: - Added ack from Christian König to the DRM patch. - Dropped "fixes" tag from the patches, like was requested by Christian König. The patches don't actually need a backport and merely improve the locking policy. - Dropped "reverts" from the patch titles to prevent them from auto-backporting by the stable bot based on the title. - Added r-b from Emil Velikov and placed the drm_WARN in the drm-shmem patch like he suggested in a comment to v1. - Corrected drm-shmem patch dma_resv_lock(obj->resv) inconsistently used with dma_resv_unlock(shmem->base.resv). Now shmem->base.resv variant is used for all locks/unlocks. Dmitry Osipenko (6): media: videobuf2: Don't assert held reservation lock for dma-buf mmapping dma-buf/heaps: Don't assert held reservation lock for dma-buf mmapping udmabuf: Don't assert held reservation lock for dma-buf mmapping drm: Don't assert held reservation lock for dma-buf mmapping dma-buf: Change locking policy for mmap() drm/shmem-helper: Switch to reservation lock drivers/dma-buf/dma-buf.c | 17 +- drivers/dma-buf/heaps/cma_heap.c | 3 - drivers/dma-buf/heaps/system_heap.c | 3 - drivers/dma-buf/udmabuf.c | 2 - drivers/gpu/drm/drm_gem_shmem_helper.c| 210 -- drivers/gpu/drm/drm_prime.c | 2 - drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c| 2 - drivers/gpu/drm/lima/lima_gem.c | 8 +- drivers/gpu/drm/omapdrm/omap_gem_dmabuf.c | 2 - drivers/gpu/drm/panfrost/panfrost_drv.c | 7 +- .../gpu/drm/panfrost/panfrost_gem_shrinker.c | 6 +- drivers/gpu/drm/panfrost/panfrost_mmu.c | 19 +- drivers/gpu/drm/tegra/gem.c | 2 - .../common/videobuf2/videobuf2-dma-contig.c | 3 - .../media/common/videobuf2/videobuf2-dma-sg.c | 3 - .../common/videobuf2/videobuf2-vmalloc.c | 3 - include/drm/drm_gem_shmem_helper.h| 14 +- 17 files changed, 119 insertions(+), 187 deletions(-) -- 2.40.1
Re: [Intel-gfx] [PATCH v3 6/6] drm/shmem-helper: Switch to reservation lock
On 5/22/23 16:02, Emil Velikov wrote:
>> -void drm_gem_shmem_put_pages(struct drm_gem_shmem_object *shmem)
>> +static int drm_gem_shmem_pin_locked(struct drm_gem_shmem_object *shmem)
>> +{
>> + int ret;
>> +
>> + dma_resv_assert_held(shmem->base.resv);
>> +
>> + ret = drm_gem_shmem_get_pages(shmem);
>> +
>> + return ret;
> With the assert_held in the getter, it would be less confusing to
> inline this and the unpin_locked functions.
Sorry, missed this comment earlier. The reason why it's a separate
function is because there will be another caller once we'll add the
drm-shrinker.
--
Best regards,
Dmitry
Re: [Intel-gfx] [PATCH v3 6/6] drm/shmem-helper: Switch to reservation lock
On 5/22/23 16:02, Emil Velikov wrote:
> Hi Dmitry,
>
> Saw v3 fly by, so I had a quick look. Original RB still stands,
> although I noticed a couple of non-blocking nitpicks.
>
> On Sun, 21 May 2023 at 22:00, Dmitry Osipenko
> wrote:
>
>> -static int drm_gem_shmem_get_pages_locked(struct drm_gem_shmem_object
>> *shmem)
>> +static int drm_gem_shmem_get_pages(struct drm_gem_shmem_object *shmem)
>> {
>
> Should this getter have a dma_resv_assert_held(shmem->base.resv); like
> it's put brethren?
>
>
>> -void drm_gem_shmem_put_pages(struct drm_gem_shmem_object *shmem)
>> +static int drm_gem_shmem_pin_locked(struct drm_gem_shmem_object *shmem)
>> +{
>> + int ret;
>> +
>> + dma_resv_assert_held(shmem->base.resv);
>> +
>> + ret = drm_gem_shmem_get_pages(shmem);
>> +
>> + return ret;
>
> With the assert_held in the getter, it would be less confusing to
> inline this and the unpin_locked functions.
>
>> +}
>> +
>> +static void drm_gem_shmem_unpin_locked(struct drm_gem_shmem_object *shmem)
>> {
>> - mutex_lock(>pages_lock);
>> - drm_gem_shmem_put_pages_locked(shmem);
>> - mutex_unlock(>pages_lock);
>> + dma_resv_assert_held(shmem->base.resv);
>> +
>> + drm_gem_shmem_put_pages(shmem);
>
> Side note: the putter has an assert_held so the extra one here seems quite
> odd.
>
> As said at the top - with or w/o these nitpicks, the original RB still stands.
Good catch. I actually added assert_held to get_pages(), but in a later
patch that is not part of this series.
--
Best regards,
Dmitry
[Intel-gfx] [PATCH v3 6/6] drm/shmem-helper: Switch to reservation lock
Replace all drm-shmem locks with a GEM reservation lock. This makes locks
consistent with dma-buf locking convention where importers are responsible
for holding reservation lock for all operations performed over dma-bufs,
preventing deadlock between dma-buf importers and exporters.
Suggested-by: Daniel Vetter
Acked-by: Thomas Zimmermann
Reviewed-by: Emil Velikov
Signed-off-by: Dmitry Osipenko
---
drivers/gpu/drm/drm_gem_shmem_helper.c| 208 --
drivers/gpu/drm/lima/lima_gem.c | 8 +-
drivers/gpu/drm/panfrost/panfrost_drv.c | 7 +-
.../gpu/drm/panfrost/panfrost_gem_shrinker.c | 6 +-
drivers/gpu/drm/panfrost/panfrost_mmu.c | 19 +-
include/drm/drm_gem_shmem_helper.h| 14 +-
6 files changed, 114 insertions(+), 148 deletions(-)
diff --git a/drivers/gpu/drm/drm_gem_shmem_helper.c
b/drivers/gpu/drm/drm_gem_shmem_helper.c
index 4ea6507a77e5..395942ca36fe 100644
--- a/drivers/gpu/drm/drm_gem_shmem_helper.c
+++ b/drivers/gpu/drm/drm_gem_shmem_helper.c
@@ -88,8 +88,6 @@ __drm_gem_shmem_create(struct drm_device *dev, size_t size,
bool private)
if (ret)
goto err_release;
- mutex_init(>pages_lock);
- mutex_init(>vmap_lock);
INIT_LIST_HEAD(>madv_list);
if (!private) {
@@ -141,11 +139,13 @@ void drm_gem_shmem_free(struct drm_gem_shmem_object
*shmem)
{
struct drm_gem_object *obj = >base;
- drm_WARN_ON(obj->dev, shmem->vmap_use_count);
-
if (obj->import_attach) {
drm_prime_gem_destroy(obj, shmem->sgt);
} else {
+ dma_resv_lock(shmem->base.resv, NULL);
+
+ drm_WARN_ON(obj->dev, shmem->vmap_use_count);
+
if (shmem->sgt) {
dma_unmap_sgtable(obj->dev->dev, shmem->sgt,
DMA_BIDIRECTIONAL, 0);
@@ -154,18 +154,18 @@ void drm_gem_shmem_free(struct drm_gem_shmem_object
*shmem)
}
if (shmem->pages)
drm_gem_shmem_put_pages(shmem);
- }
- drm_WARN_ON(obj->dev, shmem->pages_use_count);
+ drm_WARN_ON(obj->dev, shmem->pages_use_count);
+
+ dma_resv_unlock(shmem->base.resv);
+ }
drm_gem_object_release(obj);
- mutex_destroy(>pages_lock);
- mutex_destroy(>vmap_lock);
kfree(shmem);
}
EXPORT_SYMBOL_GPL(drm_gem_shmem_free);
-static int drm_gem_shmem_get_pages_locked(struct drm_gem_shmem_object *shmem)
+static int drm_gem_shmem_get_pages(struct drm_gem_shmem_object *shmem)
{
struct drm_gem_object *obj = >base;
struct page **pages;
@@ -197,35 +197,16 @@ static int drm_gem_shmem_get_pages_locked(struct
drm_gem_shmem_object *shmem)
}
/*
- * drm_gem_shmem_get_pages - Allocate backing pages for a shmem GEM object
+ * drm_gem_shmem_put_pages - Decrease use count on the backing pages for a
shmem GEM object
* @shmem: shmem GEM object
*
- * This function makes sure that backing pages exists for the shmem GEM object
- * and increases the use count.
- *
- * Returns:
- * 0 on success or a negative error code on failure.
+ * This function decreases the use count and puts the backing pages when use
drops to zero.
*/
-int drm_gem_shmem_get_pages(struct drm_gem_shmem_object *shmem)
+void drm_gem_shmem_put_pages(struct drm_gem_shmem_object *shmem)
{
struct drm_gem_object *obj = >base;
- int ret;
-
- drm_WARN_ON(obj->dev, obj->import_attach);
-
- ret = mutex_lock_interruptible(>pages_lock);
- if (ret)
- return ret;
- ret = drm_gem_shmem_get_pages_locked(shmem);
- mutex_unlock(>pages_lock);
-
- return ret;
-}
-EXPORT_SYMBOL(drm_gem_shmem_get_pages);
-static void drm_gem_shmem_put_pages_locked(struct drm_gem_shmem_object *shmem)
-{
- struct drm_gem_object *obj = >base;
+ dma_resv_assert_held(shmem->base.resv);
if (drm_WARN_ON_ONCE(obj->dev, !shmem->pages_use_count))
return;
@@ -243,20 +224,25 @@ static void drm_gem_shmem_put_pages_locked(struct
drm_gem_shmem_object *shmem)
shmem->pages_mark_accessed_on_put);
shmem->pages = NULL;
}
+EXPORT_SYMBOL(drm_gem_shmem_put_pages);
-/*
- * drm_gem_shmem_put_pages - Decrease use count on the backing pages for a
shmem GEM object
- * @shmem: shmem GEM object
- *
- * This function decreases the use count and puts the backing pages when use
drops to zero.
- */
-void drm_gem_shmem_put_pages(struct drm_gem_shmem_object *shmem)
+static int drm_gem_shmem_pin_locked(struct drm_gem_shmem_object *shmem)
+{
+ int ret;
+
+ dma_resv_assert_held(shmem->base.resv);
+
+ ret = drm_gem_shmem_get_pages(shmem);
+
+ return ret;
+}
+
+static void drm_gem_shmem_unpin_locked(struct drm_gem_shmem_object *shmem)
{
-
[Intel-gfx] [PATCH v3 5/6] dma-buf: Change locking policy for mmap()
Change locking policy of mmap() callback, making exporters responsible
for handling dma-buf reservation locking. Previous locking policy stated
that dma-buf is locked for both importers and exporters by the dma-buf
core, which caused a deadlock problem for DRM drivers in a case of
self-imported dma-bufs which required to take the lock from the DRM
exporter side.
Reviewed-by: Emil Velikov
Signed-off-by: Dmitry Osipenko
---
drivers/dma-buf/dma-buf.c | 17 +++--
1 file changed, 3 insertions(+), 14 deletions(-)
diff --git a/drivers/dma-buf/dma-buf.c b/drivers/dma-buf/dma-buf.c
index aa4ea8530cb3..21916bba77d5 100644
--- a/drivers/dma-buf/dma-buf.c
+++ b/drivers/dma-buf/dma-buf.c
@@ -131,7 +131,6 @@ static struct file_system_type dma_buf_fs_type = {
static int dma_buf_mmap_internal(struct file *file, struct vm_area_struct *vma)
{
struct dma_buf *dmabuf;
- int ret;
if (!is_dma_buf_file(file))
return -EINVAL;
@@ -147,11 +146,7 @@ static int dma_buf_mmap_internal(struct file *file, struct
vm_area_struct *vma)
dmabuf->size >> PAGE_SHIFT)
return -EINVAL;
- dma_resv_lock(dmabuf->resv, NULL);
- ret = dmabuf->ops->mmap(dmabuf, vma);
- dma_resv_unlock(dmabuf->resv);
-
- return ret;
+ return dmabuf->ops->mmap(dmabuf, vma);
}
static loff_t dma_buf_llseek(struct file *file, loff_t offset, int whence)
@@ -850,6 +845,7 @@ static struct sg_table * __map_dma_buf(struct
dma_buf_attachment *attach,
* - _buf_ops.release()
* - _buf_ops.begin_cpu_access()
* - _buf_ops.end_cpu_access()
+ * - _buf_ops.mmap()
*
* 2. These _buf_ops callbacks are invoked with locked dma-buf
*reservation and exporter can't take the lock:
@@ -858,7 +854,6 @@ static struct sg_table * __map_dma_buf(struct
dma_buf_attachment *attach,
* - _buf_ops.unpin()
* - _buf_ops.map_dma_buf()
* - _buf_ops.unmap_dma_buf()
- * - _buf_ops.mmap()
* - _buf_ops.vmap()
* - _buf_ops.vunmap()
*
@@ -1463,8 +1458,6 @@ EXPORT_SYMBOL_NS_GPL(dma_buf_end_cpu_access, DMA_BUF);
int dma_buf_mmap(struct dma_buf *dmabuf, struct vm_area_struct *vma,
unsigned long pgoff)
{
- int ret;
-
if (WARN_ON(!dmabuf || !vma))
return -EINVAL;
@@ -1485,11 +1478,7 @@ int dma_buf_mmap(struct dma_buf *dmabuf, struct
vm_area_struct *vma,
vma_set_file(vma, dmabuf->file);
vma->vm_pgoff = pgoff;
- dma_resv_lock(dmabuf->resv, NULL);
- ret = dmabuf->ops->mmap(dmabuf, vma);
- dma_resv_unlock(dmabuf->resv);
-
- return ret;
+ return dmabuf->ops->mmap(dmabuf, vma);
}
EXPORT_SYMBOL_NS_GPL(dma_buf_mmap, DMA_BUF);
--
2.40.1
[Intel-gfx] [PATCH v3 4/6] drm: Don't assert held reservation lock for dma-buf mmapping
Don't assert held dma-buf reservation lock on memory mapping of exported buffer. We're going to change dma-buf mmap() locking policy such that exporters will have to handle the lock. The previous locking policy caused deadlock problem for DRM drivers in a case of self-imported dma-bufs once these drivers are moved to use reservation lock universally. The problem is solved by moving the lock down to exporters. This patch prepares DRM drivers for the locking policy update. Reviewed-by: Emil Velikov Acked-by: Christian König Signed-off-by: Dmitry Osipenko --- drivers/gpu/drm/drm_prime.c| 2 -- drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c | 2 -- drivers/gpu/drm/omapdrm/omap_gem_dmabuf.c | 2 -- drivers/gpu/drm/tegra/gem.c| 2 -- 4 files changed, 8 deletions(-) diff --git a/drivers/gpu/drm/drm_prime.c b/drivers/gpu/drm/drm_prime.c index d29dafce9bb0..ccb2bfb29f14 100644 --- a/drivers/gpu/drm/drm_prime.c +++ b/drivers/gpu/drm/drm_prime.c @@ -785,8 +785,6 @@ int drm_gem_dmabuf_mmap(struct dma_buf *dma_buf, struct vm_area_struct *vma) struct drm_gem_object *obj = dma_buf->priv; struct drm_device *dev = obj->dev; - dma_resv_assert_held(dma_buf->resv); - if (!dev->driver->gem_prime_mmap) return -ENOSYS; diff --git a/drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c b/drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c index fd556a076d05..1df74f7aa3dc 100644 --- a/drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c +++ b/drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c @@ -97,8 +97,6 @@ static int i915_gem_dmabuf_mmap(struct dma_buf *dma_buf, struct vm_area_struct * struct drm_i915_private *i915 = to_i915(obj->base.dev); int ret; - dma_resv_assert_held(dma_buf->resv); - if (obj->base.size < vma->vm_end - vma->vm_start) return -EINVAL; diff --git a/drivers/gpu/drm/omapdrm/omap_gem_dmabuf.c b/drivers/gpu/drm/omapdrm/omap_gem_dmabuf.c index 3abc47521b2c..8e194dbc9506 100644 --- a/drivers/gpu/drm/omapdrm/omap_gem_dmabuf.c +++ b/drivers/gpu/drm/omapdrm/omap_gem_dmabuf.c @@ -66,8 +66,6 @@ static int omap_gem_dmabuf_mmap(struct dma_buf *buffer, struct drm_gem_object *obj = buffer->priv; int ret = 0; - dma_resv_assert_held(buffer->resv); - ret = drm_gem_mmap_obj(obj, omap_gem_mmap_size(obj), vma); if (ret < 0) return ret; diff --git a/drivers/gpu/drm/tegra/gem.c b/drivers/gpu/drm/tegra/gem.c index dea38892d6e6..a4023163493d 100644 --- a/drivers/gpu/drm/tegra/gem.c +++ b/drivers/gpu/drm/tegra/gem.c @@ -694,8 +694,6 @@ static int tegra_gem_prime_mmap(struct dma_buf *buf, struct vm_area_struct *vma) struct drm_gem_object *gem = buf->priv; int err; - dma_resv_assert_held(buf->resv); - err = drm_gem_mmap_obj(gem, gem->size, vma); if (err < 0) return err; -- 2.40.1
[Intel-gfx] [PATCH v3 2/6] dma-buf/heaps: Don't assert held reservation lock for dma-buf mmapping
Don't assert held dma-buf reservation lock on memory mapping of exported
buffer.
We're going to change dma-buf mmap() locking policy such that exporters
will have to handle the lock. The previous locking policy caused deadlock
problem for DRM drivers in a case of self-imported dma-bufs once these
drivers are moved to use reservation lock universally. The problem
solved by moving the lock down to exporters. This patch prepares dma-buf
heaps for the locking policy update.
Reviewed-by: Emil Velikov
Signed-off-by: Dmitry Osipenko
---
drivers/dma-buf/heaps/cma_heap.c| 3 ---
drivers/dma-buf/heaps/system_heap.c | 3 ---
2 files changed, 6 deletions(-)
diff --git a/drivers/dma-buf/heaps/cma_heap.c b/drivers/dma-buf/heaps/cma_heap.c
index a7f048048864..ee899f8e6721 100644
--- a/drivers/dma-buf/heaps/cma_heap.c
+++ b/drivers/dma-buf/heaps/cma_heap.c
@@ -13,7 +13,6 @@
#include
#include
#include
-#include
#include
#include
#include
@@ -183,8 +182,6 @@ static int cma_heap_mmap(struct dma_buf *dmabuf, struct
vm_area_struct *vma)
{
struct cma_heap_buffer *buffer = dmabuf->priv;
- dma_resv_assert_held(dmabuf->resv);
-
if ((vma->vm_flags & (VM_SHARED | VM_MAYSHARE)) == 0)
return -EINVAL;
diff --git a/drivers/dma-buf/heaps/system_heap.c
b/drivers/dma-buf/heaps/system_heap.c
index ee7059399e9c..9076d47ed2ef 100644
--- a/drivers/dma-buf/heaps/system_heap.c
+++ b/drivers/dma-buf/heaps/system_heap.c
@@ -13,7 +13,6 @@
#include
#include
#include
-#include
#include
#include
#include
@@ -201,8 +200,6 @@ static int system_heap_mmap(struct dma_buf *dmabuf, struct
vm_area_struct *vma)
struct sg_page_iter piter;
int ret;
- dma_resv_assert_held(dmabuf->resv);
-
for_each_sgtable_page(table, , vma->vm_pgoff) {
struct page *page = sg_page_iter_page();
--
2.40.1
[Intel-gfx] [PATCH v3 1/6] media: videobuf2: Don't assert held reservation lock for dma-buf mmapping
Don't assert held dma-buf reservation lock on memory mapping of exported
buffer.
We're going to change dma-buf mmap() locking policy such that exporters
will have to handle the lock. The previous locking policy caused deadlock
problem for DRM drivers in a case of self-imported dma-bufs once these
drivers are moved to use reservation lock universally. The problem is
solved by moving the lock down to exporters. This patch prepares videobuf2
for the locking policy update.
Reviewed-by: Emil Velikov
Reviewed-by: Hans Verkuil
Signed-off-by: Dmitry Osipenko
---
drivers/media/common/videobuf2/videobuf2-dma-contig.c | 3 ---
drivers/media/common/videobuf2/videobuf2-dma-sg.c | 3 ---
drivers/media/common/videobuf2/videobuf2-vmalloc.c| 3 ---
3 files changed, 9 deletions(-)
diff --git a/drivers/media/common/videobuf2/videobuf2-dma-contig.c
b/drivers/media/common/videobuf2/videobuf2-dma-contig.c
index 205d3cac425c..2fa455d4a048 100644
--- a/drivers/media/common/videobuf2/videobuf2-dma-contig.c
+++ b/drivers/media/common/videobuf2/videobuf2-dma-contig.c
@@ -11,7 +11,6 @@
*/
#include
-#include
#include
#include
#include
@@ -456,8 +455,6 @@ static int vb2_dc_dmabuf_ops_vmap(struct dma_buf *dbuf,
struct iosys_map *map)
static int vb2_dc_dmabuf_ops_mmap(struct dma_buf *dbuf,
struct vm_area_struct *vma)
{
- dma_resv_assert_held(dbuf->resv);
-
return vb2_dc_mmap(dbuf->priv, vma);
}
diff --git a/drivers/media/common/videobuf2/videobuf2-dma-sg.c
b/drivers/media/common/videobuf2/videobuf2-dma-sg.c
index 183037fb1273..28f3fdfe23a2 100644
--- a/drivers/media/common/videobuf2/videobuf2-dma-sg.c
+++ b/drivers/media/common/videobuf2/videobuf2-dma-sg.c
@@ -10,7 +10,6 @@
* the Free Software Foundation.
*/
-#include
#include
#include
#include
@@ -498,8 +497,6 @@ static int vb2_dma_sg_dmabuf_ops_vmap(struct dma_buf *dbuf,
static int vb2_dma_sg_dmabuf_ops_mmap(struct dma_buf *dbuf,
struct vm_area_struct *vma)
{
- dma_resv_assert_held(dbuf->resv);
-
return vb2_dma_sg_mmap(dbuf->priv, vma);
}
diff --git a/drivers/media/common/videobuf2/videobuf2-vmalloc.c
b/drivers/media/common/videobuf2/videobuf2-vmalloc.c
index a6c6d2fcaaa4..7c635e292106 100644
--- a/drivers/media/common/videobuf2/videobuf2-vmalloc.c
+++ b/drivers/media/common/videobuf2/videobuf2-vmalloc.c
@@ -10,7 +10,6 @@
* the Free Software Foundation.
*/
-#include
#include
#include
#include
@@ -319,8 +318,6 @@ static int vb2_vmalloc_dmabuf_ops_vmap(struct dma_buf *dbuf,
static int vb2_vmalloc_dmabuf_ops_mmap(struct dma_buf *dbuf,
struct vm_area_struct *vma)
{
- dma_resv_assert_held(dbuf->resv);
-
return vb2_vmalloc_mmap(dbuf->priv, vma);
}
--
2.40.1
[Intel-gfx] [PATCH v3 3/6] udmabuf: Don't assert held reservation lock for dma-buf mmapping
Don't assert held dma-buf reservation lock on memory mapping of exported
buffer.
We're going to change dma-buf mmap() locking policy such that exporters
will have to handle the lock. The previous locking policy caused deadlock
problem for DRM drivers in a case of self-imported dma-bufs once these
drivers are moved to use reservation lock universally. The problem is
solved by moving the lock down to exporters. This patch prepares udmabuf
for the locking policy update.
Reviewed-by: Emil Velikov
Signed-off-by: Dmitry Osipenko
---
drivers/dma-buf/udmabuf.c | 2 --
1 file changed, 2 deletions(-)
diff --git a/drivers/dma-buf/udmabuf.c b/drivers/dma-buf/udmabuf.c
index 01f2e86f3f7c..06729cd60136 100644
--- a/drivers/dma-buf/udmabuf.c
+++ b/drivers/dma-buf/udmabuf.c
@@ -52,8 +52,6 @@ static int mmap_udmabuf(struct dma_buf *buf, struct
vm_area_struct *vma)
{
struct udmabuf *ubuf = buf->priv;
- dma_resv_assert_held(buf->resv);
-
if ((vma->vm_flags & (VM_SHARED | VM_MAYSHARE)) == 0)
return -EINVAL;
--
2.40.1
[Intel-gfx] [PATCH v3 0/6] Move dma-buf mmap() reservation locking down to exporters
This patchset makes dma-buf exporters responisble for taking care of the reservation lock. I also included patch that moves drm-shmem to use reservation lock, to let CI test the whole set. I'm going to take all the patches via the drm-misc tree, please give an ack. Previous policy stated that dma-buf core takes the lock around mmap() callback. Which meant that both importers and exporters shouldn't touch the reservation lock in the mmap() code path. This worked well until Intel-CI found a deadlock problem in a case of self-imported dma-buf [1]. The problem happens when userpace mmaps a self-imported dma-buf, i.e. mmaps the dma-buf FD. DRM core treats self-imported dma-bufs as own GEMs [2]. There is no way to differentiate a prime GEM from a normal GEM for drm-shmem in drm_gem_shmem_mmap(), which resulted in a deadlock problem for drm-shmem mmap() code path once it's switched to use reservation lock. It was difficult to fix the drm-shmem problem without adjusting dma-buf locking policy. In parctice not much changed from importers perspective because previosly dma-buf was taking the lock in between of importers and exporters. Now this lock is shifted down to exporters. [1] https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_114671v2/shard-snb5/igt@prime_vgem@s...@rcs0.html [2] https://elixir.bootlin.com/linux/v6.3-rc4/source/drivers/gpu/drm/drm_prime.c#L924 Changelog: v3: - Added r-b from Hans Verkuil to the videobuf2 patch. - The v2 fastrpc patch was already applied, not including it anymore. Though, the cover-letter says that I'd want to apply all the patches via the drm-misc tree to keep the proper ordering of the changes. - Previously Intel's CI gave a flake failure to v2, want to re-test it again. v2: - Added ack from Christian König to the DRM patch. - Dropped "fixes" tag from the patches, like was requested by Christian König. The patches don't actually need a backport and merely improve the locking policy. - Dropped "reverts" from the patch titles to prevent them from auto-backporting by the stable bot based on the title. - Added r-b from Emil Velikov and placed the drm_WARN in the drm-shmem patch like he suggested in a comment to v1. - Corrected drm-shmem patch dma_resv_lock(obj->resv) inconsistently used with dma_resv_unlock(shmem->base.resv). Now shmem->base.resv variant is used for all locks/unlocks. Dmitry Osipenko (6): media: videobuf2: Don't assert held reservation lock for dma-buf mmapping dma-buf/heaps: Don't assert held reservation lock for dma-buf mmapping udmabuf: Don't assert held reservation lock for dma-buf mmapping drm: Don't assert held reservation lock for dma-buf mmapping dma-buf: Change locking policy for mmap() drm/shmem-helper: Switch to reservation lock drivers/dma-buf/dma-buf.c | 17 +- drivers/dma-buf/heaps/cma_heap.c | 3 - drivers/dma-buf/heaps/system_heap.c | 3 - drivers/dma-buf/udmabuf.c | 2 - drivers/gpu/drm/drm_gem_shmem_helper.c| 208 -- drivers/gpu/drm/drm_prime.c | 2 - drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c| 2 - drivers/gpu/drm/lima/lima_gem.c | 8 +- drivers/gpu/drm/omapdrm/omap_gem_dmabuf.c | 2 - drivers/gpu/drm/panfrost/panfrost_drv.c | 7 +- .../gpu/drm/panfrost/panfrost_gem_shrinker.c | 6 +- drivers/gpu/drm/panfrost/panfrost_mmu.c | 19 +- drivers/gpu/drm/tegra/gem.c | 2 - .../common/videobuf2/videobuf2-dma-contig.c | 3 - .../media/common/videobuf2/videobuf2-dma-sg.c | 3 - .../common/videobuf2/videobuf2-vmalloc.c | 3 - include/drm/drm_gem_shmem_helper.h| 14 +- 17 files changed, 117 insertions(+), 187 deletions(-) -- 2.40.1
Re: [Intel-gfx] [PATCH v2 6/7] dma-buf: Change locking policy for mmap()
On 4/6/23 19:06, Dmitry Osipenko wrote: > Change locking policy of mmap() callback, making exporters responsible > for handling dma-buf reservation locking. Previous locking policy stated > that dma-buf is locked for both importers and exporters by the dma-buf > core, which caused a deadlock problem for DRM drivers in a case of > self-imported dma-bufs which required to take the lock from the DRM > exporter side. > > Reviewed-by: Emil Velikov > Signed-off-by: Dmitry Osipenko > --- > drivers/dma-buf/dma-buf.c | 17 +++-- > 1 file changed, 3 insertions(+), 14 deletions(-) Christian, may I add yours ack to this patch? -- Best regards, Dmitry
[Intel-gfx] [PATCH v2 2/7] dma-buf/heaps: Don't assert held reservation lock for dma-buf mmapping
Don't assert held dma-buf reservation lock on memory mapping of exported
buffer.
We're going to change dma-buf mmap() locking policy such that exporters
will have to handle the lock. The previous locking policy caused deadlock
problem for DRM drivers in a case of self-imported dma-bufs once these
drivers are moved to use reservation lock universally. The problem
solved by moving the lock down to exporters. This patch prepares dma-buf
heaps for the locking policy update.
Reviewed-by: Emil Velikov
Signed-off-by: Dmitry Osipenko
---
drivers/dma-buf/heaps/cma_heap.c| 3 ---
drivers/dma-buf/heaps/system_heap.c | 3 ---
2 files changed, 6 deletions(-)
diff --git a/drivers/dma-buf/heaps/cma_heap.c b/drivers/dma-buf/heaps/cma_heap.c
index 1131fb943992..28fb04eccdd0 100644
--- a/drivers/dma-buf/heaps/cma_heap.c
+++ b/drivers/dma-buf/heaps/cma_heap.c
@@ -13,7 +13,6 @@
#include
#include
#include
-#include
#include
#include
#include
@@ -183,8 +182,6 @@ static int cma_heap_mmap(struct dma_buf *dmabuf, struct
vm_area_struct *vma)
{
struct cma_heap_buffer *buffer = dmabuf->priv;
- dma_resv_assert_held(dmabuf->resv);
-
if ((vma->vm_flags & (VM_SHARED | VM_MAYSHARE)) == 0)
return -EINVAL;
diff --git a/drivers/dma-buf/heaps/system_heap.c
b/drivers/dma-buf/heaps/system_heap.c
index e8bd10e60998..fcf836ba9c1f 100644
--- a/drivers/dma-buf/heaps/system_heap.c
+++ b/drivers/dma-buf/heaps/system_heap.c
@@ -13,7 +13,6 @@
#include
#include
#include
-#include
#include
#include
#include
@@ -202,8 +201,6 @@ static int system_heap_mmap(struct dma_buf *dmabuf, struct
vm_area_struct *vma)
struct sg_page_iter piter;
int ret;
- dma_resv_assert_held(dmabuf->resv);
-
for_each_sgtable_page(table, , vma->vm_pgoff) {
struct page *page = sg_page_iter_page();
--
2.39.2
[Intel-gfx] [PATCH v2 6/7] dma-buf: Change locking policy for mmap()
Change locking policy of mmap() callback, making exporters responsible
for handling dma-buf reservation locking. Previous locking policy stated
that dma-buf is locked for both importers and exporters by the dma-buf
core, which caused a deadlock problem for DRM drivers in a case of
self-imported dma-bufs which required to take the lock from the DRM
exporter side.
Reviewed-by: Emil Velikov
Signed-off-by: Dmitry Osipenko
---
drivers/dma-buf/dma-buf.c | 17 +++--
1 file changed, 3 insertions(+), 14 deletions(-)
diff --git a/drivers/dma-buf/dma-buf.c b/drivers/dma-buf/dma-buf.c
index aa4ea8530cb3..21916bba77d5 100644
--- a/drivers/dma-buf/dma-buf.c
+++ b/drivers/dma-buf/dma-buf.c
@@ -131,7 +131,6 @@ static struct file_system_type dma_buf_fs_type = {
static int dma_buf_mmap_internal(struct file *file, struct vm_area_struct *vma)
{
struct dma_buf *dmabuf;
- int ret;
if (!is_dma_buf_file(file))
return -EINVAL;
@@ -147,11 +146,7 @@ static int dma_buf_mmap_internal(struct file *file, struct
vm_area_struct *vma)
dmabuf->size >> PAGE_SHIFT)
return -EINVAL;
- dma_resv_lock(dmabuf->resv, NULL);
- ret = dmabuf->ops->mmap(dmabuf, vma);
- dma_resv_unlock(dmabuf->resv);
-
- return ret;
+ return dmabuf->ops->mmap(dmabuf, vma);
}
static loff_t dma_buf_llseek(struct file *file, loff_t offset, int whence)
@@ -850,6 +845,7 @@ static struct sg_table * __map_dma_buf(struct
dma_buf_attachment *attach,
* - _buf_ops.release()
* - _buf_ops.begin_cpu_access()
* - _buf_ops.end_cpu_access()
+ * - _buf_ops.mmap()
*
* 2. These _buf_ops callbacks are invoked with locked dma-buf
*reservation and exporter can't take the lock:
@@ -858,7 +854,6 @@ static struct sg_table * __map_dma_buf(struct
dma_buf_attachment *attach,
* - _buf_ops.unpin()
* - _buf_ops.map_dma_buf()
* - _buf_ops.unmap_dma_buf()
- * - _buf_ops.mmap()
* - _buf_ops.vmap()
* - _buf_ops.vunmap()
*
@@ -1463,8 +1458,6 @@ EXPORT_SYMBOL_NS_GPL(dma_buf_end_cpu_access, DMA_BUF);
int dma_buf_mmap(struct dma_buf *dmabuf, struct vm_area_struct *vma,
unsigned long pgoff)
{
- int ret;
-
if (WARN_ON(!dmabuf || !vma))
return -EINVAL;
@@ -1485,11 +1478,7 @@ int dma_buf_mmap(struct dma_buf *dmabuf, struct
vm_area_struct *vma,
vma_set_file(vma, dmabuf->file);
vma->vm_pgoff = pgoff;
- dma_resv_lock(dmabuf->resv, NULL);
- ret = dmabuf->ops->mmap(dmabuf, vma);
- dma_resv_unlock(dmabuf->resv);
-
- return ret;
+ return dmabuf->ops->mmap(dmabuf, vma);
}
EXPORT_SYMBOL_NS_GPL(dma_buf_mmap, DMA_BUF);
--
2.39.2
[Intel-gfx] [PATCH v2 7/7] drm/shmem-helper: Switch to reservation lock
Replace all drm-shmem locks with a GEM reservation lock. This makes locks
consistent with dma-buf locking convention where importers are responsible
for holding reservation lock for all operations performed over dma-bufs,
preventing deadlock between dma-buf importers and exporters.
Suggested-by: Daniel Vetter
Acked-by: Thomas Zimmermann
Reviewed-by: Emil Velikov
Signed-off-by: Dmitry Osipenko
---
drivers/gpu/drm/drm_gem_shmem_helper.c| 208 --
drivers/gpu/drm/lima/lima_gem.c | 8 +-
drivers/gpu/drm/panfrost/panfrost_drv.c | 7 +-
.../gpu/drm/panfrost/panfrost_gem_shrinker.c | 6 +-
drivers/gpu/drm/panfrost/panfrost_mmu.c | 19 +-
include/drm/drm_gem_shmem_helper.h| 14 +-
6 files changed, 114 insertions(+), 148 deletions(-)
diff --git a/drivers/gpu/drm/drm_gem_shmem_helper.c
b/drivers/gpu/drm/drm_gem_shmem_helper.c
index 4ea6507a77e5..395942ca36fe 100644
--- a/drivers/gpu/drm/drm_gem_shmem_helper.c
+++ b/drivers/gpu/drm/drm_gem_shmem_helper.c
@@ -88,8 +88,6 @@ __drm_gem_shmem_create(struct drm_device *dev, size_t size,
bool private)
if (ret)
goto err_release;
- mutex_init(>pages_lock);
- mutex_init(>vmap_lock);
INIT_LIST_HEAD(>madv_list);
if (!private) {
@@ -141,11 +139,13 @@ void drm_gem_shmem_free(struct drm_gem_shmem_object
*shmem)
{
struct drm_gem_object *obj = >base;
- drm_WARN_ON(obj->dev, shmem->vmap_use_count);
-
if (obj->import_attach) {
drm_prime_gem_destroy(obj, shmem->sgt);
} else {
+ dma_resv_lock(shmem->base.resv, NULL);
+
+ drm_WARN_ON(obj->dev, shmem->vmap_use_count);
+
if (shmem->sgt) {
dma_unmap_sgtable(obj->dev->dev, shmem->sgt,
DMA_BIDIRECTIONAL, 0);
@@ -154,18 +154,18 @@ void drm_gem_shmem_free(struct drm_gem_shmem_object
*shmem)
}
if (shmem->pages)
drm_gem_shmem_put_pages(shmem);
- }
- drm_WARN_ON(obj->dev, shmem->pages_use_count);
+ drm_WARN_ON(obj->dev, shmem->pages_use_count);
+
+ dma_resv_unlock(shmem->base.resv);
+ }
drm_gem_object_release(obj);
- mutex_destroy(>pages_lock);
- mutex_destroy(>vmap_lock);
kfree(shmem);
}
EXPORT_SYMBOL_GPL(drm_gem_shmem_free);
-static int drm_gem_shmem_get_pages_locked(struct drm_gem_shmem_object *shmem)
+static int drm_gem_shmem_get_pages(struct drm_gem_shmem_object *shmem)
{
struct drm_gem_object *obj = >base;
struct page **pages;
@@ -197,35 +197,16 @@ static int drm_gem_shmem_get_pages_locked(struct
drm_gem_shmem_object *shmem)
}
/*
- * drm_gem_shmem_get_pages - Allocate backing pages for a shmem GEM object
+ * drm_gem_shmem_put_pages - Decrease use count on the backing pages for a
shmem GEM object
* @shmem: shmem GEM object
*
- * This function makes sure that backing pages exists for the shmem GEM object
- * and increases the use count.
- *
- * Returns:
- * 0 on success or a negative error code on failure.
+ * This function decreases the use count and puts the backing pages when use
drops to zero.
*/
-int drm_gem_shmem_get_pages(struct drm_gem_shmem_object *shmem)
+void drm_gem_shmem_put_pages(struct drm_gem_shmem_object *shmem)
{
struct drm_gem_object *obj = >base;
- int ret;
-
- drm_WARN_ON(obj->dev, obj->import_attach);
-
- ret = mutex_lock_interruptible(>pages_lock);
- if (ret)
- return ret;
- ret = drm_gem_shmem_get_pages_locked(shmem);
- mutex_unlock(>pages_lock);
-
- return ret;
-}
-EXPORT_SYMBOL(drm_gem_shmem_get_pages);
-static void drm_gem_shmem_put_pages_locked(struct drm_gem_shmem_object *shmem)
-{
- struct drm_gem_object *obj = >base;
+ dma_resv_assert_held(shmem->base.resv);
if (drm_WARN_ON_ONCE(obj->dev, !shmem->pages_use_count))
return;
@@ -243,20 +224,25 @@ static void drm_gem_shmem_put_pages_locked(struct
drm_gem_shmem_object *shmem)
shmem->pages_mark_accessed_on_put);
shmem->pages = NULL;
}
+EXPORT_SYMBOL(drm_gem_shmem_put_pages);
-/*
- * drm_gem_shmem_put_pages - Decrease use count on the backing pages for a
shmem GEM object
- * @shmem: shmem GEM object
- *
- * This function decreases the use count and puts the backing pages when use
drops to zero.
- */
-void drm_gem_shmem_put_pages(struct drm_gem_shmem_object *shmem)
+static int drm_gem_shmem_pin_locked(struct drm_gem_shmem_object *shmem)
+{
+ int ret;
+
+ dma_resv_assert_held(shmem->base.resv);
+
+ ret = drm_gem_shmem_get_pages(shmem);
+
+ return ret;
+}
+
+static void drm_gem_shmem_unpin_locked(struct drm_gem_shmem_object *shmem)
{
-
[Intel-gfx] [PATCH v2 5/7] drm: Don't assert held reservation lock for dma-buf mmapping
Don't assert held dma-buf reservation lock on memory mapping of exported buffer. We're going to change dma-buf mmap() locking policy such that exporters will have to handle the lock. The previous locking policy caused deadlock problem for DRM drivers in a case of self-imported dma-bufs once these drivers are moved to use reservation lock universally. The problem is solved by moving the lock down to exporters. This patch prepares DRM drivers for the locking policy update. Reviewed-by: Emil Velikov Acked-by: Christian König Signed-off-by: Dmitry Osipenko --- drivers/gpu/drm/drm_prime.c| 2 -- drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c | 2 -- drivers/gpu/drm/omapdrm/omap_gem_dmabuf.c | 2 -- drivers/gpu/drm/tegra/gem.c| 2 -- 4 files changed, 8 deletions(-) diff --git a/drivers/gpu/drm/drm_prime.c b/drivers/gpu/drm/drm_prime.c index 149cd4ff6a3b..cea85e84666f 100644 --- a/drivers/gpu/drm/drm_prime.c +++ b/drivers/gpu/drm/drm_prime.c @@ -781,8 +781,6 @@ int drm_gem_dmabuf_mmap(struct dma_buf *dma_buf, struct vm_area_struct *vma) struct drm_gem_object *obj = dma_buf->priv; struct drm_device *dev = obj->dev; - dma_resv_assert_held(dma_buf->resv); - if (!dev->driver->gem_prime_mmap) return -ENOSYS; diff --git a/drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c b/drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c index fd556a076d05..1df74f7aa3dc 100644 --- a/drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c +++ b/drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c @@ -97,8 +97,6 @@ static int i915_gem_dmabuf_mmap(struct dma_buf *dma_buf, struct vm_area_struct * struct drm_i915_private *i915 = to_i915(obj->base.dev); int ret; - dma_resv_assert_held(dma_buf->resv); - if (obj->base.size < vma->vm_end - vma->vm_start) return -EINVAL; diff --git a/drivers/gpu/drm/omapdrm/omap_gem_dmabuf.c b/drivers/gpu/drm/omapdrm/omap_gem_dmabuf.c index 3abc47521b2c..8e194dbc9506 100644 --- a/drivers/gpu/drm/omapdrm/omap_gem_dmabuf.c +++ b/drivers/gpu/drm/omapdrm/omap_gem_dmabuf.c @@ -66,8 +66,6 @@ static int omap_gem_dmabuf_mmap(struct dma_buf *buffer, struct drm_gem_object *obj = buffer->priv; int ret = 0; - dma_resv_assert_held(buffer->resv); - ret = drm_gem_mmap_obj(obj, omap_gem_mmap_size(obj), vma); if (ret < 0) return ret; diff --git a/drivers/gpu/drm/tegra/gem.c b/drivers/gpu/drm/tegra/gem.c index bce991a2ccc0..871ef5d26523 100644 --- a/drivers/gpu/drm/tegra/gem.c +++ b/drivers/gpu/drm/tegra/gem.c @@ -693,8 +693,6 @@ static int tegra_gem_prime_mmap(struct dma_buf *buf, struct vm_area_struct *vma) struct drm_gem_object *gem = buf->priv; int err; - dma_resv_assert_held(buf->resv); - err = drm_gem_mmap_obj(gem, gem->size, vma); if (err < 0) return err; -- 2.39.2
[Intel-gfx] [PATCH v2 4/7] fastrpc: Don't assert held reservation lock for dma-buf mmapping
Don't assert held dma-buf reservation lock on memory mapping of exported buffer. We're going to change dma-buf mmap() locking policy such that exporters will have to handle the lock. The previous locking policy caused deadlock problem for DRM drivers in a case of self-imported dma-bufs once these drivers are moved to use reservation lock universally. The problem is solved by moving the lock down to exporters. This patch prepares fastrpc for the locking policy update. Reviewed-by: Emil Velikov Signed-off-by: Dmitry Osipenko --- drivers/misc/fastrpc.c | 3 --- 1 file changed, 3 deletions(-) diff --git a/drivers/misc/fastrpc.c b/drivers/misc/fastrpc.c index a701132638cf..7e9c9ad37fd9 100644 --- a/drivers/misc/fastrpc.c +++ b/drivers/misc/fastrpc.c @@ -6,7 +6,6 @@ #include #include #include -#include #include #include #include @@ -733,8 +732,6 @@ static int fastrpc_mmap(struct dma_buf *dmabuf, struct fastrpc_buf *buf = dmabuf->priv; size_t size = vma->vm_end - vma->vm_start; - dma_resv_assert_held(dmabuf->resv); - return dma_mmap_coherent(buf->dev, vma, buf->virt, FASTRPC_PHYS(buf->phys), size); } -- 2.39.2
[Intel-gfx] [PATCH v2 3/7] udmabuf: Don't assert held reservation lock for dma-buf mmapping
Don't assert held dma-buf reservation lock on memory mapping of exported
buffer.
We're going to change dma-buf mmap() locking policy such that exporters
will have to handle the lock. The previous locking policy caused deadlock
problem for DRM drivers in a case of self-imported dma-bufs once these
drivers are moved to use reservation lock universally. The problem is
solved by moving the lock down to exporters. This patch prepares udmabuf
for the locking policy update.
Reviewed-by: Emil Velikov
Signed-off-by: Dmitry Osipenko
---
drivers/dma-buf/udmabuf.c | 2 --
1 file changed, 2 deletions(-)
diff --git a/drivers/dma-buf/udmabuf.c b/drivers/dma-buf/udmabuf.c
index 740d6e426ee9..277f1afa9552 100644
--- a/drivers/dma-buf/udmabuf.c
+++ b/drivers/dma-buf/udmabuf.c
@@ -52,8 +52,6 @@ static int mmap_udmabuf(struct dma_buf *buf, struct
vm_area_struct *vma)
{
struct udmabuf *ubuf = buf->priv;
- dma_resv_assert_held(buf->resv);
-
if ((vma->vm_flags & (VM_SHARED | VM_MAYSHARE)) == 0)
return -EINVAL;
--
2.39.2
[Intel-gfx] [PATCH v2 1/7] media: videobuf2: Don't assert held reservation lock for dma-buf mmapping
Don't assert held dma-buf reservation lock on memory mapping of exported
buffer.
We're going to change dma-buf mmap() locking policy such that exporters
will have to handle the lock. The previous locking policy caused deadlock
problem for DRM drivers in a case of self-imported dma-bufs once these
drivers are moved to use reservation lock universally. The problem is
solved by moving the lock down to exporters. This patch prepares videobuf2
for the locking policy update.
Reviewed-by: Emil Velikov
Signed-off-by: Dmitry Osipenko
---
drivers/media/common/videobuf2/videobuf2-dma-contig.c | 3 ---
drivers/media/common/videobuf2/videobuf2-dma-sg.c | 3 ---
drivers/media/common/videobuf2/videobuf2-vmalloc.c| 3 ---
3 files changed, 9 deletions(-)
diff --git a/drivers/media/common/videobuf2/videobuf2-dma-contig.c
b/drivers/media/common/videobuf2/videobuf2-dma-contig.c
index 205d3cac425c..2fa455d4a048 100644
--- a/drivers/media/common/videobuf2/videobuf2-dma-contig.c
+++ b/drivers/media/common/videobuf2/videobuf2-dma-contig.c
@@ -11,7 +11,6 @@
*/
#include
-#include
#include
#include
#include
@@ -456,8 +455,6 @@ static int vb2_dc_dmabuf_ops_vmap(struct dma_buf *dbuf,
struct iosys_map *map)
static int vb2_dc_dmabuf_ops_mmap(struct dma_buf *dbuf,
struct vm_area_struct *vma)
{
- dma_resv_assert_held(dbuf->resv);
-
return vb2_dc_mmap(dbuf->priv, vma);
}
diff --git a/drivers/media/common/videobuf2/videobuf2-dma-sg.c
b/drivers/media/common/videobuf2/videobuf2-dma-sg.c
index 183037fb1273..28f3fdfe23a2 100644
--- a/drivers/media/common/videobuf2/videobuf2-dma-sg.c
+++ b/drivers/media/common/videobuf2/videobuf2-dma-sg.c
@@ -10,7 +10,6 @@
* the Free Software Foundation.
*/
-#include
#include
#include
#include
@@ -498,8 +497,6 @@ static int vb2_dma_sg_dmabuf_ops_vmap(struct dma_buf *dbuf,
static int vb2_dma_sg_dmabuf_ops_mmap(struct dma_buf *dbuf,
struct vm_area_struct *vma)
{
- dma_resv_assert_held(dbuf->resv);
-
return vb2_dma_sg_mmap(dbuf->priv, vma);
}
diff --git a/drivers/media/common/videobuf2/videobuf2-vmalloc.c
b/drivers/media/common/videobuf2/videobuf2-vmalloc.c
index a6c6d2fcaaa4..7c635e292106 100644
--- a/drivers/media/common/videobuf2/videobuf2-vmalloc.c
+++ b/drivers/media/common/videobuf2/videobuf2-vmalloc.c
@@ -10,7 +10,6 @@
* the Free Software Foundation.
*/
-#include
#include
#include
#include
@@ -319,8 +318,6 @@ static int vb2_vmalloc_dmabuf_ops_vmap(struct dma_buf *dbuf,
static int vb2_vmalloc_dmabuf_ops_mmap(struct dma_buf *dbuf,
struct vm_area_struct *vma)
{
- dma_resv_assert_held(dbuf->resv);
-
return vb2_vmalloc_mmap(dbuf->priv, vma);
}
--
2.39.2
[Intel-gfx] [PATCH v2 0/7] Move dma-buf mmap() reservation locking down to exporters
This patchset makes dma-buf exporters responisble for taking care of the reservation lock. I also included patch that moves drm-shmem to use reservation lock, to let CI test the whole set. I'm going to take all the patches via the drm-misc tree, please give an ack. Previous policy stated that dma-buf core takes the lock around mmap() callback. Which meant that both importers and exporters shouldn't touch the reservation lock in the mmap() code path. This worked well until Intel-CI found a deadlock problem in a case of self-imported dma-buf [1]. The problem happens when userpace mmaps a self-imported dma-buf, i.e. mmaps the dma-buf FD. DRM core treats self-imported dma-bufs as own GEMs [2]. There is no way to differentiate a prime GEM from a normal GEM for drm-shmem in drm_gem_shmem_mmap(), which resulted in a deadlock problem for drm-shmem mmap() code path once it's switched to use reservation lock. It was difficult to fix the drm-shmem problem without adjusting dma-buf locking policy. In parctice not much changed from importers perspective because previosly dma-buf was taking the lock in between of importers and exporters. Now this lock is shifted down to exporters. [1] https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_114671v2/shard-snb5/igt@prime_vgem@s...@rcs0.html [2] https://elixir.bootlin.com/linux/v6.3-rc4/source/drivers/gpu/drm/drm_prime.c#L924 Changelog: v2: - Added ack from Christian König to the DRM patch. - Dropped "fixes" tag from the patches, like was requested by Christian König. The patches don't actually need a backport and merely improve the locking policy. - Dropped "reverts" from the patch titles to prevent them from auto-backporting by the stable bot based on the title. - Added r-b from Emil Velikov and placed the drm_WARN in the drm-shmem patch like he suggested in a comment to v1. - Corrected drm-shmem patch dma_resv_lock(obj->resv) inconsistently used with dma_resv_unlock(shmem->base.resv). Now shmem->base.resv variant is used for all locks/unlocks. Dmitry Osipenko (7): media: videobuf2: Don't assert held reservation lock for dma-buf mmapping dma-buf/heaps: Don't assert held reservation lock for dma-buf mmapping udmabuf: Don't assert held reservation lock for dma-buf mmapping fastrpc: Don't assert held reservation lock for dma-buf mmapping drm: Don't assert held reservation lock for dma-buf mmapping dma-buf: Change locking policy for mmap() drm/shmem-helper: Switch to reservation lock drivers/dma-buf/dma-buf.c | 17 +- drivers/dma-buf/heaps/cma_heap.c | 3 - drivers/dma-buf/heaps/system_heap.c | 3 - drivers/dma-buf/udmabuf.c | 2 - drivers/gpu/drm/drm_gem_shmem_helper.c| 208 -- drivers/gpu/drm/drm_prime.c | 2 - drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c| 2 - drivers/gpu/drm/lima/lima_gem.c | 8 +- drivers/gpu/drm/omapdrm/omap_gem_dmabuf.c | 2 - drivers/gpu/drm/panfrost/panfrost_drv.c | 7 +- .../gpu/drm/panfrost/panfrost_gem_shrinker.c | 6 +- drivers/gpu/drm/panfrost/panfrost_mmu.c | 19 +- drivers/gpu/drm/tegra/gem.c | 2 - .../common/videobuf2/videobuf2-dma-contig.c | 3 - .../media/common/videobuf2/videobuf2-dma-sg.c | 3 - .../common/videobuf2/videobuf2-vmalloc.c | 3 - drivers/misc/fastrpc.c| 3 - include/drm/drm_gem_shmem_helper.h| 14 +- 18 files changed, 117 insertions(+), 190 deletions(-) -- 2.39.2
Re: [Intel-gfx] [PATCH v1 5/7] Revert "drm: Assert held reservation lock for dma-buf mmapping"
On 4/3/23 18:17, Christian König wrote:
> Am 02.04.23 um 18:48 schrieb Dmitry Osipenko:
>> Don't assert held dma-buf reservation lock on memory mapping of exported
>> buffer.
>>
>> We're going to change dma-buf mmap() locking policy such that exporters
>> will have to handle the lock. The previous locking policy caused deadlock
>> problem for DRM drivers in a case of self-imported dma-bufs, it's solved
>> by moving the lock down to exporters.
>
> I only checked the TTM code path and think that at least that one should
> work fine.
>
>> Fixes: 39ce25291871 ("drm: Assert held reservation lock for dma-buf
>> mmapping")
>
> This here is not really a "fix" for the previous patch. We just found
> that we didn't like the behavior and so reverted the original patch.
>
> A "Reverts..." comment in the commit message would be more appropriate I
> think.
Ack, will drop the fixes tag in v2. Thank you and Emil for the review.
--
Best regards,
Dmitry
Re: [Intel-gfx] [PATCH v1 0/7] Move dma-buf mmap() reservation locking down to exporters
On 4/2/23 19:48, Dmitry Osipenko wrote: > This patchset makes dma-buf exporters responisble for taking care of > the reservation lock. I also included patch that moves drm-shmem to use > reservation lock, to let CI test the whole set. I'm going to take all > the patches via the drm-misc tree, please give an ack. > > Previous policy stated that dma-buf core takes the lock around mmap() > callback. Which meant that both importers and exporters shouldn't touch > the reservation lock in the mmap() code path. This worked well until > Intel-CI found a deadlock problem in a case of self-imported dma-buf [1]. > > The problem happens when userpace mmaps a self-imported dma-buf, i.e. > mmaps the dma-buf FD. DRM core treats self-imported dma-bufs as own GEMs > [2]. There is no way to differentiate a prime GEM from a normal GEM for > drm-shmem in drm_gem_shmem_mmap(), which resulted in a deadlock problem > for drm-shmem mmap() code path once it's switched to use reservation lock. > > It was difficult to fix the drm-shmem problem without adjusting dma-buf > locking policy. In parctice not much changed from importers perspective > because previosly dma-buf was taking the lock in between of importers > and exporters. Now this lock is shifted down to exporters. > > [1] > https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_114671v2/shard-snb5/igt@prime_vgem@s...@rcs0.html > [2] > https://elixir.bootlin.com/linux/v6.3-rc4/source/drivers/gpu/drm/drm_prime.c#L924 > > Dmitry Osipenko (7): > Revert "media: videobuf2: Assert held reservation lock for dma-buf > mmapping" > Revert "dma-buf/heaps: Assert held reservation lock for dma-buf > mmapping" > Revert "udmabuf: Assert held reservation lock for dma-buf mmapping" > Revert "fastrpc: Assert held reservation lock for dma-buf mmapping" > Revert "drm: Assert held reservation lock for dma-buf mmapping" > dma-buf: Change locking policy for mmap() > drm/shmem-helper: Switch to reservation lock > > drivers/dma-buf/dma-buf.c | 17 +- > drivers/dma-buf/heaps/cma_heap.c | 3 - > drivers/dma-buf/heaps/system_heap.c | 3 - > drivers/dma-buf/udmabuf.c | 2 - > drivers/gpu/drm/drm_gem_shmem_helper.c| 217 -- > drivers/gpu/drm/drm_prime.c | 2 - > drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c| 2 - > drivers/gpu/drm/lima/lima_gem.c | 8 +- > drivers/gpu/drm/omapdrm/omap_gem_dmabuf.c | 2 - > drivers/gpu/drm/panfrost/panfrost_drv.c | 7 +- > .../gpu/drm/panfrost/panfrost_gem_shrinker.c | 6 +- > drivers/gpu/drm/panfrost/panfrost_mmu.c | 19 +- > drivers/gpu/drm/tegra/gem.c | 2 - > .../common/videobuf2/videobuf2-dma-contig.c | 3 - > .../media/common/videobuf2/videobuf2-dma-sg.c | 3 - > .../common/videobuf2/videobuf2-vmalloc.c | 3 - > drivers/misc/fastrpc.c| 3 - > include/drm/drm_gem_shmem_helper.h| 14 +- > 18 files changed, 123 insertions(+), 193 deletions(-) > Intel's IGT passed[1] (see Tests section) with the new locking policy, which failed previously for the "drm/shmem-helper: Switch to reservation lock" patch. [1] https://patchwork.freedesktop.org/series/116000/ -- Best regards, Dmitry
[Intel-gfx] [PATCH v1 7/7] drm/shmem-helper: Switch to reservation lock
Replace all drm-shmem locks with a GEM reservation lock. This makes locks
consistent with dma-buf locking convention where importers are responsible
for holding reservation lock for all operations performed over dma-bufs,
preventing deadlock between dma-buf importers and exporters.
Suggested-by: Daniel Vetter
Acked-by: Thomas Zimmermann
Signed-off-by: Dmitry Osipenko
---
drivers/gpu/drm/drm_gem_shmem_helper.c| 217 --
drivers/gpu/drm/lima/lima_gem.c | 8 +-
drivers/gpu/drm/panfrost/panfrost_drv.c | 7 +-
.../gpu/drm/panfrost/panfrost_gem_shrinker.c | 6 +-
drivers/gpu/drm/panfrost/panfrost_mmu.c | 19 +-
include/drm/drm_gem_shmem_helper.h| 14 +-
6 files changed, 120 insertions(+), 151 deletions(-)
diff --git a/drivers/gpu/drm/drm_gem_shmem_helper.c
b/drivers/gpu/drm/drm_gem_shmem_helper.c
index 4ea6507a77e5..8fc2a3277486 100644
--- a/drivers/gpu/drm/drm_gem_shmem_helper.c
+++ b/drivers/gpu/drm/drm_gem_shmem_helper.c
@@ -88,8 +88,6 @@ __drm_gem_shmem_create(struct drm_device *dev, size_t size,
bool private)
if (ret)
goto err_release;
- mutex_init(>pages_lock);
- mutex_init(>vmap_lock);
INIT_LIST_HEAD(>madv_list);
if (!private) {
@@ -141,11 +139,13 @@ void drm_gem_shmem_free(struct drm_gem_shmem_object
*shmem)
{
struct drm_gem_object *obj = >base;
- drm_WARN_ON(obj->dev, shmem->vmap_use_count);
-
if (obj->import_attach) {
drm_prime_gem_destroy(obj, shmem->sgt);
} else {
+ dma_resv_lock(shmem->base.resv, NULL);
+
+ drm_WARN_ON(obj->dev, shmem->vmap_use_count);
+
if (shmem->sgt) {
dma_unmap_sgtable(obj->dev->dev, shmem->sgt,
DMA_BIDIRECTIONAL, 0);
@@ -154,18 +154,18 @@ void drm_gem_shmem_free(struct drm_gem_shmem_object
*shmem)
}
if (shmem->pages)
drm_gem_shmem_put_pages(shmem);
- }
- drm_WARN_ON(obj->dev, shmem->pages_use_count);
+ drm_WARN_ON(obj->dev, shmem->pages_use_count);
+
+ dma_resv_unlock(shmem->base.resv);
+ }
drm_gem_object_release(obj);
- mutex_destroy(>pages_lock);
- mutex_destroy(>vmap_lock);
kfree(shmem);
}
EXPORT_SYMBOL_GPL(drm_gem_shmem_free);
-static int drm_gem_shmem_get_pages_locked(struct drm_gem_shmem_object *shmem)
+static int drm_gem_shmem_get_pages(struct drm_gem_shmem_object *shmem)
{
struct drm_gem_object *obj = >base;
struct page **pages;
@@ -197,35 +197,16 @@ static int drm_gem_shmem_get_pages_locked(struct
drm_gem_shmem_object *shmem)
}
/*
- * drm_gem_shmem_get_pages - Allocate backing pages for a shmem GEM object
+ * drm_gem_shmem_put_pages - Decrease use count on the backing pages for a
shmem GEM object
* @shmem: shmem GEM object
*
- * This function makes sure that backing pages exists for the shmem GEM object
- * and increases the use count.
- *
- * Returns:
- * 0 on success or a negative error code on failure.
+ * This function decreases the use count and puts the backing pages when use
drops to zero.
*/
-int drm_gem_shmem_get_pages(struct drm_gem_shmem_object *shmem)
+void drm_gem_shmem_put_pages(struct drm_gem_shmem_object *shmem)
{
struct drm_gem_object *obj = >base;
- int ret;
- drm_WARN_ON(obj->dev, obj->import_attach);
-
- ret = mutex_lock_interruptible(>pages_lock);
- if (ret)
- return ret;
- ret = drm_gem_shmem_get_pages_locked(shmem);
- mutex_unlock(>pages_lock);
-
- return ret;
-}
-EXPORT_SYMBOL(drm_gem_shmem_get_pages);
-
-static void drm_gem_shmem_put_pages_locked(struct drm_gem_shmem_object *shmem)
-{
- struct drm_gem_object *obj = >base;
+ dma_resv_assert_held(shmem->base.resv);
if (drm_WARN_ON_ONCE(obj->dev, !shmem->pages_use_count))
return;
@@ -243,20 +224,32 @@ static void drm_gem_shmem_put_pages_locked(struct
drm_gem_shmem_object *shmem)
shmem->pages_mark_accessed_on_put);
shmem->pages = NULL;
}
+EXPORT_SYMBOL(drm_gem_shmem_put_pages);
-/*
- * drm_gem_shmem_put_pages - Decrease use count on the backing pages for a
shmem GEM object
- * @shmem: shmem GEM object
- *
- * This function decreases the use count and puts the backing pages when use
drops to zero.
- */
-void drm_gem_shmem_put_pages(struct drm_gem_shmem_object *shmem)
+static int drm_gem_shmem_pin_locked(struct drm_gem_shmem_object *shmem)
{
- mutex_lock(>pages_lock);
- drm_gem_shmem_put_pages_locked(shmem);
- mutex_unlock(>pages_lock);
+ struct drm_gem_object *obj = >base;
+ int ret;
+
+ dma_resv_assert_held(shmem->base.resv);
+
+ drm_WARN_ON(o
[Intel-gfx] [PATCH v1 6/7] dma-buf: Change locking policy for mmap()
Change locking policy of mmap() callback, making exporters responsible
for handling dma-buf reservation locking. Previous locking policy stated
that dma-buf is locked for both importers and exporters by the dma-buf
core, which caused a deadlock problem for DRM drivers in a case of
self-imported dma-bufs which required to take the lock from the DRM
exporter side.
Signed-off-by: Dmitry Osipenko
---
drivers/dma-buf/dma-buf.c | 17 +++--
1 file changed, 3 insertions(+), 14 deletions(-)
diff --git a/drivers/dma-buf/dma-buf.c b/drivers/dma-buf/dma-buf.c
index aa4ea8530cb3..21916bba77d5 100644
--- a/drivers/dma-buf/dma-buf.c
+++ b/drivers/dma-buf/dma-buf.c
@@ -131,7 +131,6 @@ static struct file_system_type dma_buf_fs_type = {
static int dma_buf_mmap_internal(struct file *file, struct vm_area_struct *vma)
{
struct dma_buf *dmabuf;
- int ret;
if (!is_dma_buf_file(file))
return -EINVAL;
@@ -147,11 +146,7 @@ static int dma_buf_mmap_internal(struct file *file, struct
vm_area_struct *vma)
dmabuf->size >> PAGE_SHIFT)
return -EINVAL;
- dma_resv_lock(dmabuf->resv, NULL);
- ret = dmabuf->ops->mmap(dmabuf, vma);
- dma_resv_unlock(dmabuf->resv);
-
- return ret;
+ return dmabuf->ops->mmap(dmabuf, vma);
}
static loff_t dma_buf_llseek(struct file *file, loff_t offset, int whence)
@@ -850,6 +845,7 @@ static struct sg_table * __map_dma_buf(struct
dma_buf_attachment *attach,
* - _buf_ops.release()
* - _buf_ops.begin_cpu_access()
* - _buf_ops.end_cpu_access()
+ * - _buf_ops.mmap()
*
* 2. These _buf_ops callbacks are invoked with locked dma-buf
*reservation and exporter can't take the lock:
@@ -858,7 +854,6 @@ static struct sg_table * __map_dma_buf(struct
dma_buf_attachment *attach,
* - _buf_ops.unpin()
* - _buf_ops.map_dma_buf()
* - _buf_ops.unmap_dma_buf()
- * - _buf_ops.mmap()
* - _buf_ops.vmap()
* - _buf_ops.vunmap()
*
@@ -1463,8 +1458,6 @@ EXPORT_SYMBOL_NS_GPL(dma_buf_end_cpu_access, DMA_BUF);
int dma_buf_mmap(struct dma_buf *dmabuf, struct vm_area_struct *vma,
unsigned long pgoff)
{
- int ret;
-
if (WARN_ON(!dmabuf || !vma))
return -EINVAL;
@@ -1485,11 +1478,7 @@ int dma_buf_mmap(struct dma_buf *dmabuf, struct
vm_area_struct *vma,
vma_set_file(vma, dmabuf->file);
vma->vm_pgoff = pgoff;
- dma_resv_lock(dmabuf->resv, NULL);
- ret = dmabuf->ops->mmap(dmabuf, vma);
- dma_resv_unlock(dmabuf->resv);
-
- return ret;
+ return dmabuf->ops->mmap(dmabuf, vma);
}
EXPORT_SYMBOL_NS_GPL(dma_buf_mmap, DMA_BUF);
--
2.39.2
[Intel-gfx] [PATCH v1 5/7] Revert "drm: Assert held reservation lock for dma-buf mmapping"
Don't assert held dma-buf reservation lock on memory mapping of exported
buffer.
We're going to change dma-buf mmap() locking policy such that exporters
will have to handle the lock. The previous locking policy caused deadlock
problem for DRM drivers in a case of self-imported dma-bufs, it's solved
by moving the lock down to exporters.
Fixes: 39ce25291871 ("drm: Assert held reservation lock for dma-buf mmapping")
Signed-off-by: Dmitry Osipenko
---
drivers/gpu/drm/drm_prime.c| 2 --
drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c | 2 --
drivers/gpu/drm/omapdrm/omap_gem_dmabuf.c | 2 --
drivers/gpu/drm/tegra/gem.c| 2 --
4 files changed, 8 deletions(-)
diff --git a/drivers/gpu/drm/drm_prime.c b/drivers/gpu/drm/drm_prime.c
index 149cd4ff6a3b..cea85e84666f 100644
--- a/drivers/gpu/drm/drm_prime.c
+++ b/drivers/gpu/drm/drm_prime.c
@@ -781,8 +781,6 @@ int drm_gem_dmabuf_mmap(struct dma_buf *dma_buf, struct
vm_area_struct *vma)
struct drm_gem_object *obj = dma_buf->priv;
struct drm_device *dev = obj->dev;
- dma_resv_assert_held(dma_buf->resv);
-
if (!dev->driver->gem_prime_mmap)
return -ENOSYS;
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c
b/drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c
index fd556a076d05..1df74f7aa3dc 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c
@@ -97,8 +97,6 @@ static int i915_gem_dmabuf_mmap(struct dma_buf *dma_buf,
struct vm_area_struct *
struct drm_i915_private *i915 = to_i915(obj->base.dev);
int ret;
- dma_resv_assert_held(dma_buf->resv);
-
if (obj->base.size < vma->vm_end - vma->vm_start)
return -EINVAL;
diff --git a/drivers/gpu/drm/omapdrm/omap_gem_dmabuf.c
b/drivers/gpu/drm/omapdrm/omap_gem_dmabuf.c
index 3abc47521b2c..8e194dbc9506 100644
--- a/drivers/gpu/drm/omapdrm/omap_gem_dmabuf.c
+++ b/drivers/gpu/drm/omapdrm/omap_gem_dmabuf.c
@@ -66,8 +66,6 @@ static int omap_gem_dmabuf_mmap(struct dma_buf *buffer,
struct drm_gem_object *obj = buffer->priv;
int ret = 0;
- dma_resv_assert_held(buffer->resv);
-
ret = drm_gem_mmap_obj(obj, omap_gem_mmap_size(obj), vma);
if (ret < 0)
return ret;
diff --git a/drivers/gpu/drm/tegra/gem.c b/drivers/gpu/drm/tegra/gem.c
index bce991a2ccc0..871ef5d26523 100644
--- a/drivers/gpu/drm/tegra/gem.c
+++ b/drivers/gpu/drm/tegra/gem.c
@@ -693,8 +693,6 @@ static int tegra_gem_prime_mmap(struct dma_buf *buf, struct
vm_area_struct *vma)
struct drm_gem_object *gem = buf->priv;
int err;
- dma_resv_assert_held(buf->resv);
-
err = drm_gem_mmap_obj(gem, gem->size, vma);
if (err < 0)
return err;
--
2.39.2
[Intel-gfx] [PATCH v1 4/7] Revert "fastrpc: Assert held reservation lock for dma-buf mmapping"
Don't assert held dma-buf reservation lock on memory mapping of exported
buffer.
We're going to change dma-buf mmap() locking policy such that exporters
will have to handle the lock. The previous locking policy caused deadlock
problem for DRM drivers in a case of self-imported dma-bufs, it's solved
by moving the lock down to exporters.
Fixes: 265751a513ad ("fastrpc: Assert held reservation lock for dma-buf
mmapping")
Signed-off-by: Dmitry Osipenko
---
drivers/misc/fastrpc.c | 3 ---
1 file changed, 3 deletions(-)
diff --git a/drivers/misc/fastrpc.c b/drivers/misc/fastrpc.c
index a701132638cf..7e9c9ad37fd9 100644
--- a/drivers/misc/fastrpc.c
+++ b/drivers/misc/fastrpc.c
@@ -6,7 +6,6 @@
#include
#include
#include
-#include
#include
#include
#include
@@ -733,8 +732,6 @@ static int fastrpc_mmap(struct dma_buf *dmabuf,
struct fastrpc_buf *buf = dmabuf->priv;
size_t size = vma->vm_end - vma->vm_start;
- dma_resv_assert_held(dmabuf->resv);
-
return dma_mmap_coherent(buf->dev, vma, buf->virt,
FASTRPC_PHYS(buf->phys), size);
}
--
2.39.2
[Intel-gfx] [PATCH v1 3/7] Revert "udmabuf: Assert held reservation lock for dma-buf mmapping"
Don't assert held dma-buf reservation lock on memory mapping of exported
buffer.
We're going to change dma-buf mmap() locking policy such that exporters
will have to handle the lock. The previous locking policy caused deadlock
problem for DRM drivers in a case of self-imported dma-bufs, it's solved
by moving the lock down to exporters.
Fixes: aa3f99896443 ("udmabuf: Assert held reservation lock for dma-buf
mmapping")
Signed-off-by: Dmitry Osipenko
---
drivers/dma-buf/udmabuf.c | 2 --
1 file changed, 2 deletions(-)
diff --git a/drivers/dma-buf/udmabuf.c b/drivers/dma-buf/udmabuf.c
index 740d6e426ee9..277f1afa9552 100644
--- a/drivers/dma-buf/udmabuf.c
+++ b/drivers/dma-buf/udmabuf.c
@@ -52,8 +52,6 @@ static int mmap_udmabuf(struct dma_buf *buf, struct
vm_area_struct *vma)
{
struct udmabuf *ubuf = buf->priv;
- dma_resv_assert_held(buf->resv);
-
if ((vma->vm_flags & (VM_SHARED | VM_MAYSHARE)) == 0)
return -EINVAL;
--
2.39.2
[Intel-gfx] [PATCH v1 2/7] Revert "dma-buf/heaps: Assert held reservation lock for dma-buf mmapping"
Don't assert held dma-buf reservation lock on memory mapping of exported
buffer.
We're going to change dma-buf mmap() locking policy such that exporters
will have to handle the lock. The previous locking policy caused deadlock
problem for DRM drivers in a case of self-imported dma-bufs, it's solved
by moving the lock down to exporters.
Fixes: 27f3733a1049 ("dma-buf/heaps: Assert held reservation lock for dma-buf
mmapping")
Signed-off-by: Dmitry Osipenko
---
drivers/dma-buf/heaps/cma_heap.c| 3 ---
drivers/dma-buf/heaps/system_heap.c | 3 ---
2 files changed, 6 deletions(-)
diff --git a/drivers/dma-buf/heaps/cma_heap.c b/drivers/dma-buf/heaps/cma_heap.c
index 1131fb943992..28fb04eccdd0 100644
--- a/drivers/dma-buf/heaps/cma_heap.c
+++ b/drivers/dma-buf/heaps/cma_heap.c
@@ -13,7 +13,6 @@
#include
#include
#include
-#include
#include
#include
#include
@@ -183,8 +182,6 @@ static int cma_heap_mmap(struct dma_buf *dmabuf, struct
vm_area_struct *vma)
{
struct cma_heap_buffer *buffer = dmabuf->priv;
- dma_resv_assert_held(dmabuf->resv);
-
if ((vma->vm_flags & (VM_SHARED | VM_MAYSHARE)) == 0)
return -EINVAL;
diff --git a/drivers/dma-buf/heaps/system_heap.c
b/drivers/dma-buf/heaps/system_heap.c
index e8bd10e60998..fcf836ba9c1f 100644
--- a/drivers/dma-buf/heaps/system_heap.c
+++ b/drivers/dma-buf/heaps/system_heap.c
@@ -13,7 +13,6 @@
#include
#include
#include
-#include
#include
#include
#include
@@ -202,8 +201,6 @@ static int system_heap_mmap(struct dma_buf *dmabuf, struct
vm_area_struct *vma)
struct sg_page_iter piter;
int ret;
- dma_resv_assert_held(dmabuf->resv);
-
for_each_sgtable_page(table, , vma->vm_pgoff) {
struct page *page = sg_page_iter_page();
--
2.39.2
[Intel-gfx] [PATCH v1 1/7] Revert "media: videobuf2: Assert held reservation lock for dma-buf mmapping"
Don't assert held dma-buf reservation lock on memory mapping of exported
buffer.
We're going to change dma-buf mmap() locking policy such that exporters
will have to handle the lock. The previous locking policy caused deadlock
problem for DRM drivers in a case of self-imported dma-bufs, it's solved
by moving the lock down to exporters.
Fixes: 3a6ca1810f77 ("media: videobuf2: Assert held reservation lock for
dma-buf mmapping")
Signed-off-by: Dmitry Osipenko
---
drivers/media/common/videobuf2/videobuf2-dma-contig.c | 3 ---
drivers/media/common/videobuf2/videobuf2-dma-sg.c | 3 ---
drivers/media/common/videobuf2/videobuf2-vmalloc.c| 3 ---
3 files changed, 9 deletions(-)
diff --git a/drivers/media/common/videobuf2/videobuf2-dma-contig.c
b/drivers/media/common/videobuf2/videobuf2-dma-contig.c
index 205d3cac425c..2fa455d4a048 100644
--- a/drivers/media/common/videobuf2/videobuf2-dma-contig.c
+++ b/drivers/media/common/videobuf2/videobuf2-dma-contig.c
@@ -11,7 +11,6 @@
*/
#include
-#include
#include
#include
#include
@@ -456,8 +455,6 @@ static int vb2_dc_dmabuf_ops_vmap(struct dma_buf *dbuf,
struct iosys_map *map)
static int vb2_dc_dmabuf_ops_mmap(struct dma_buf *dbuf,
struct vm_area_struct *vma)
{
- dma_resv_assert_held(dbuf->resv);
-
return vb2_dc_mmap(dbuf->priv, vma);
}
diff --git a/drivers/media/common/videobuf2/videobuf2-dma-sg.c
b/drivers/media/common/videobuf2/videobuf2-dma-sg.c
index 183037fb1273..28f3fdfe23a2 100644
--- a/drivers/media/common/videobuf2/videobuf2-dma-sg.c
+++ b/drivers/media/common/videobuf2/videobuf2-dma-sg.c
@@ -10,7 +10,6 @@
* the Free Software Foundation.
*/
-#include
#include
#include
#include
@@ -498,8 +497,6 @@ static int vb2_dma_sg_dmabuf_ops_vmap(struct dma_buf *dbuf,
static int vb2_dma_sg_dmabuf_ops_mmap(struct dma_buf *dbuf,
struct vm_area_struct *vma)
{
- dma_resv_assert_held(dbuf->resv);
-
return vb2_dma_sg_mmap(dbuf->priv, vma);
}
diff --git a/drivers/media/common/videobuf2/videobuf2-vmalloc.c
b/drivers/media/common/videobuf2/videobuf2-vmalloc.c
index a6c6d2fcaaa4..7c635e292106 100644
--- a/drivers/media/common/videobuf2/videobuf2-vmalloc.c
+++ b/drivers/media/common/videobuf2/videobuf2-vmalloc.c
@@ -10,7 +10,6 @@
* the Free Software Foundation.
*/
-#include
#include
#include
#include
@@ -319,8 +318,6 @@ static int vb2_vmalloc_dmabuf_ops_vmap(struct dma_buf *dbuf,
static int vb2_vmalloc_dmabuf_ops_mmap(struct dma_buf *dbuf,
struct vm_area_struct *vma)
{
- dma_resv_assert_held(dbuf->resv);
-
return vb2_vmalloc_mmap(dbuf->priv, vma);
}
--
2.39.2
[Intel-gfx] [PATCH v1 0/7] Move dma-buf mmap() reservation locking down to exporters
This patchset makes dma-buf exporters responisble for taking care of the reservation lock. I also included patch that moves drm-shmem to use reservation lock, to let CI test the whole set. I'm going to take all the patches via the drm-misc tree, please give an ack. Previous policy stated that dma-buf core takes the lock around mmap() callback. Which meant that both importers and exporters shouldn't touch the reservation lock in the mmap() code path. This worked well until Intel-CI found a deadlock problem in a case of self-imported dma-buf [1]. The problem happens when userpace mmaps a self-imported dma-buf, i.e. mmaps the dma-buf FD. DRM core treats self-imported dma-bufs as own GEMs [2]. There is no way to differentiate a prime GEM from a normal GEM for drm-shmem in drm_gem_shmem_mmap(), which resulted in a deadlock problem for drm-shmem mmap() code path once it's switched to use reservation lock. It was difficult to fix the drm-shmem problem without adjusting dma-buf locking policy. In parctice not much changed from importers perspective because previosly dma-buf was taking the lock in between of importers and exporters. Now this lock is shifted down to exporters. [1] https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_114671v2/shard-snb5/igt@prime_vgem@s...@rcs0.html [2] https://elixir.bootlin.com/linux/v6.3-rc4/source/drivers/gpu/drm/drm_prime.c#L924 Dmitry Osipenko (7): Revert "media: videobuf2: Assert held reservation lock for dma-buf mmapping" Revert "dma-buf/heaps: Assert held reservation lock for dma-buf mmapping" Revert "udmabuf: Assert held reservation lock for dma-buf mmapping" Revert "fastrpc: Assert held reservation lock for dma-buf mmapping" Revert "drm: Assert held reservation lock for dma-buf mmapping" dma-buf: Change locking policy for mmap() drm/shmem-helper: Switch to reservation lock drivers/dma-buf/dma-buf.c | 17 +- drivers/dma-buf/heaps/cma_heap.c | 3 - drivers/dma-buf/heaps/system_heap.c | 3 - drivers/dma-buf/udmabuf.c | 2 - drivers/gpu/drm/drm_gem_shmem_helper.c| 217 -- drivers/gpu/drm/drm_prime.c | 2 - drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c| 2 - drivers/gpu/drm/lima/lima_gem.c | 8 +- drivers/gpu/drm/omapdrm/omap_gem_dmabuf.c | 2 - drivers/gpu/drm/panfrost/panfrost_drv.c | 7 +- .../gpu/drm/panfrost/panfrost_gem_shrinker.c | 6 +- drivers/gpu/drm/panfrost/panfrost_mmu.c | 19 +- drivers/gpu/drm/tegra/gem.c | 2 - .../common/videobuf2/videobuf2-dma-contig.c | 3 - .../media/common/videobuf2/videobuf2-dma-sg.c | 3 - .../common/videobuf2/videobuf2-vmalloc.c | 3 - drivers/misc/fastrpc.c| 3 - include/drm/drm_gem_shmem_helper.h| 14 +- 18 files changed, 123 insertions(+), 193 deletions(-) -- 2.39.2
Re: [Intel-gfx] [PATCH v13 01/10] drm/shmem-helper: Switch to reservation lock
On 3/26/23 12:19, Christian König wrote: > Am 25.03.23 um 15:58 schrieb Dmitry Osipenko: >> On 3/15/23 16:46, Dmitry Osipenko wrote: >>> On 3/14/23 05:26, Dmitry Osipenko wrote: >>>> @@ -633,7 +605,10 @@ int drm_gem_shmem_mmap(struct >>>> drm_gem_shmem_object *shmem, struct vm_area_struct >>>> return ret; >>>> } >>>> + dma_resv_lock(shmem->base.resv, NULL); >>>> ret = drm_gem_shmem_get_pages(shmem); >>>> + dma_resv_unlock(shmem->base.resv); >>> Intel CI reported locking problem [1] here. It actually was also >>> reported for v12, but I missed that report because of the other noisy >>> reports. >>> >>> [1] >>> https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_114671v2/shard-snb5/igt@prime_vgem@s...@rcs0.html >>> >>> The test does the following: >>> >>> 1. creates vgem >>> 2. export it do dmabuf >>> 3. mmaps dmabuf >>> >>> There is an obvious deadlock there. The DRM code assumes that mmap() is >>> unlocked, while for dma-buf it's locked. I'll see how to fix it for v14. >>> >> Christian, there is a deadlock problem in drm_gem_shmem_mmap() once we >> move drm-shmem to use resv lock. The current dma-buf locking policy >> claims that importer holds the lock for mmap(), but DRM code assumes >> that obj->mmap() handles the locking itself and then the same >> obj->mmap() code path is used by both dma-buf DRM and a usual DRM object >> paths. Hence importer -> dma_buf_mmap_internal()[takes the lock] -> >> exporter -> drm_gem_shmem_mmap()[takes the lock] deadlocks. >> >> I was looking at how to fix it and to me the best option is to change >> the dma-buf locking policy, making exporter responsible for handling the >> resv lock. Changing DRM code mmap lockings might be possible too [moving >> locking to drm_gem_mmap_obj()], but will be very messy and doesn't feel >> intuitive. >> >> Want to get yours thoughts on this before sending out the dma-buf mmap() >> policy-change patch. Does the new mmap() locking policy sound good to >> you? Thanks! > > > IIRC we tried that before and ran into problems. > > dma_buf_mmap() needs to swap the backing file of the VMA and for this > calls fput() on the old file. > > This fput() in turn could (in theory) grab the resv lock as well and > there isn't anything we could do about that. > > Just information from the back of my memory, probably best if you double > check that. Thanks, Christian! The fput() code path will be unlocked with updated locking policy, like it was before. The new locking policy looks goods on my side, don't see anything that needs locking protection from the importer side for mmap(). I'll send the patches, letting intel-ci test them. Will be also easier to discuss it there with the code. -- Best regards, Dmitry
Re: [Intel-gfx] [PATCH v13 01/10] drm/shmem-helper: Switch to reservation lock
On 3/15/23 16:46, Dmitry Osipenko wrote: > On 3/14/23 05:26, Dmitry Osipenko wrote: >> @@ -633,7 +605,10 @@ int drm_gem_shmem_mmap(struct drm_gem_shmem_object >> *shmem, struct vm_area_struct >> return ret; >> } >> >> +dma_resv_lock(shmem->base.resv, NULL); >> ret = drm_gem_shmem_get_pages(shmem); >> +dma_resv_unlock(shmem->base.resv); > > Intel CI reported locking problem [1] here. It actually was also > reported for v12, but I missed that report because of the other noisy > reports. > > [1] > https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_114671v2/shard-snb5/igt@prime_vgem@s...@rcs0.html > > The test does the following: > > 1. creates vgem > 2. export it do dmabuf > 3. mmaps dmabuf > > There is an obvious deadlock there. The DRM code assumes that mmap() is > unlocked, while for dma-buf it's locked. I'll see how to fix it for v14. > Christian, there is a deadlock problem in drm_gem_shmem_mmap() once we move drm-shmem to use resv lock. The current dma-buf locking policy claims that importer holds the lock for mmap(), but DRM code assumes that obj->mmap() handles the locking itself and then the same obj->mmap() code path is used by both dma-buf DRM and a usual DRM object paths. Hence importer -> dma_buf_mmap_internal()[takes the lock] -> exporter -> drm_gem_shmem_mmap()[takes the lock] deadlocks. I was looking at how to fix it and to me the best option is to change the dma-buf locking policy, making exporter responsible for handling the resv lock. Changing DRM code mmap lockings might be possible too [moving locking to drm_gem_mmap_obj()], but will be very messy and doesn't feel intuitive. Want to get yours thoughts on this before sending out the dma-buf mmap() policy-change patch. Does the new mmap() locking policy sound good to you? Thanks! -- Best regards, Dmitry
Re: [Intel-gfx] [PATCH v13 01/10] drm/shmem-helper: Switch to reservation lock
On 3/14/23 05:26, Dmitry Osipenko wrote: > @@ -633,7 +605,10 @@ int drm_gem_shmem_mmap(struct drm_gem_shmem_object > *shmem, struct vm_area_struct > return ret; > } > > + dma_resv_lock(shmem->base.resv, NULL); > ret = drm_gem_shmem_get_pages(shmem); > + dma_resv_unlock(shmem->base.resv); Intel CI reported locking problem [1] here. It actually was also reported for v12, but I missed that report because of the other noisy reports. [1] https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_114671v2/shard-snb5/igt@prime_vgem@s...@rcs0.html The test does the following: 1. creates vgem 2. export it do dmabuf 3. mmaps dmabuf There is an obvious deadlock there. The DRM code assumes that mmap() is unlocked, while for dma-buf it's locked. I'll see how to fix it for v14. -- Best regards, Dmitry
[Intel-gfx] [PATCH v13 10/10] drm/panfrost: Switch to generic memory shrinker
Replace Panfrost's custom memory shrinker with a common drm-shmem
memory shrinker.
Tested-by: Steven Price # Firefly-RK3288
Reviewed-by: Steven Price
Signed-off-by: Dmitry Osipenko
---
drivers/gpu/drm/panfrost/Makefile | 1 -
drivers/gpu/drm/panfrost/panfrost_device.h| 4 -
drivers/gpu/drm/panfrost/panfrost_drv.c | 27 ++--
drivers/gpu/drm/panfrost/panfrost_gem.c | 30 ++--
drivers/gpu/drm/panfrost/panfrost_gem.h | 9 --
.../gpu/drm/panfrost/panfrost_gem_shrinker.c | 129 --
drivers/gpu/drm/panfrost/panfrost_job.c | 18 ++-
include/drm/drm_gem_shmem_helper.h| 7 -
8 files changed, 47 insertions(+), 178 deletions(-)
delete mode 100644 drivers/gpu/drm/panfrost/panfrost_gem_shrinker.c
diff --git a/drivers/gpu/drm/panfrost/Makefile
b/drivers/gpu/drm/panfrost/Makefile
index 7da2b3f02ed9..11622e22cf15 100644
--- a/drivers/gpu/drm/panfrost/Makefile
+++ b/drivers/gpu/drm/panfrost/Makefile
@@ -5,7 +5,6 @@ panfrost-y := \
panfrost_device.o \
panfrost_devfreq.o \
panfrost_gem.o \
- panfrost_gem_shrinker.o \
panfrost_gpu.o \
panfrost_job.o \
panfrost_mmu.o \
diff --git a/drivers/gpu/drm/panfrost/panfrost_device.h
b/drivers/gpu/drm/panfrost/panfrost_device.h
index d9ba68cffb77..28f28bbdbda9 100644
--- a/drivers/gpu/drm/panfrost/panfrost_device.h
+++ b/drivers/gpu/drm/panfrost/panfrost_device.h
@@ -116,10 +116,6 @@ struct panfrost_device {
atomic_t pending;
} reset;
- struct mutex shrinker_lock;
- struct list_head shrinker_list;
- struct shrinker shrinker;
-
struct panfrost_devfreq pfdevfreq;
};
diff --git a/drivers/gpu/drm/panfrost/panfrost_drv.c
b/drivers/gpu/drm/panfrost/panfrost_drv.c
index aa292e4a86eb..e29a2e604257 100644
--- a/drivers/gpu/drm/panfrost/panfrost_drv.c
+++ b/drivers/gpu/drm/panfrost/panfrost_drv.c
@@ -169,7 +169,6 @@ panfrost_lookup_bos(struct drm_device *dev,
break;
}
- atomic_inc(>gpu_usecount);
job->mappings[i] = mapping;
}
@@ -394,7 +393,6 @@ static int panfrost_ioctl_madvise(struct drm_device *dev,
void *data,
{
struct panfrost_file_priv *priv = file_priv->driver_priv;
struct drm_panfrost_madvise *args = data;
- struct panfrost_device *pfdev = dev->dev_private;
struct drm_gem_object *gem_obj;
struct panfrost_gem_object *bo;
int ret = 0;
@@ -407,11 +405,15 @@ static int panfrost_ioctl_madvise(struct drm_device *dev,
void *data,
bo = to_panfrost_bo(gem_obj);
+ if (bo->is_heap) {
+ args->retained = 1;
+ goto out_put_object;
+ }
+
ret = dma_resv_lock_interruptible(bo->base.base.resv, NULL);
if (ret)
goto out_put_object;
- mutex_lock(>shrinker_lock);
mutex_lock(>mappings.lock);
if (args->madv == PANFROST_MADV_DONTNEED) {
struct panfrost_gem_mapping *first;
@@ -437,17 +439,8 @@ static int panfrost_ioctl_madvise(struct drm_device *dev,
void *data,
args->retained = drm_gem_shmem_madvise(>base, args->madv);
- if (args->retained) {
- if (args->madv == PANFROST_MADV_DONTNEED)
- list_move_tail(>base.madv_list,
- >shrinker_list);
- else if (args->madv == PANFROST_MADV_WILLNEED)
- list_del_init(>base.madv_list);
- }
-
out_unlock_mappings:
mutex_unlock(>mappings.lock);
- mutex_unlock(>shrinker_lock);
dma_resv_unlock(bo->base.base.resv);
out_put_object:
drm_gem_object_put(gem_obj);
@@ -579,9 +572,6 @@ static int panfrost_probe(struct platform_device *pdev)
ddev->dev_private = pfdev;
pfdev->ddev = ddev;
- mutex_init(>shrinker_lock);
- INIT_LIST_HEAD(>shrinker_list);
-
err = panfrost_device_init(pfdev);
if (err) {
if (err != -EPROBE_DEFER)
@@ -603,10 +593,14 @@ static int panfrost_probe(struct platform_device *pdev)
if (err < 0)
goto err_out1;
- panfrost_gem_shrinker_init(ddev);
+ err = drmm_gem_shmem_init(ddev);
+ if (err < 0)
+ goto err_out2;
return 0;
+err_out2:
+ drm_dev_unregister(ddev);
err_out1:
pm_runtime_disable(pfdev->dev);
panfrost_device_fini(pfdev);
@@ -622,7 +616,6 @@ static int panfrost_remove(struct platform_device *pdev)
struct drm_device *ddev = pfdev->ddev;
drm_dev_unregister(ddev);
- panfrost_gem_shrinker_cleanup(ddev);
pm_runtime_get_sync(pfdev->dev);
pm_runtime_disable(pfdev->dev);
diff --git a/drivers/gpu/drm/panfrost/panfrost_gem.c
b/drivers/gpu/drm/panfrost/panfrost_gem.c
index 3c812
[Intel-gfx] [PATCH v13 09/10] drm/virtio: Support memory shrinking
Support generic drm-shmem memory shrinker and add new madvise IOCTL to
the VirtIO-GPU driver. BO cache manager of Mesa driver will mark BOs as
"don't need" using the new IOCTL to let shrinker purge the marked BOs on
OOM, the shrinker will also evict unpurgeable shmem BOs from memory if
guest supports SWAP file or partition.
Acked-by: Gerd Hoffmann
Signed-off-by: Daniel Almeida
Signed-off-by: Dmitry Osipenko
---
drivers/gpu/drm/virtio/virtgpu_drv.h| 20 +++-
drivers/gpu/drm/virtio/virtgpu_gem.c| 68
drivers/gpu/drm/virtio/virtgpu_ioctl.c | 37 +++
drivers/gpu/drm/virtio/virtgpu_kms.c| 8 ++
drivers/gpu/drm/virtio/virtgpu_object.c | 137 +++-
drivers/gpu/drm/virtio/virtgpu_plane.c | 17 ++-
drivers/gpu/drm/virtio/virtgpu_vq.c | 40 +++
include/uapi/drm/virtgpu_drm.h | 14 +++
8 files changed, 310 insertions(+), 31 deletions(-)
diff --git a/drivers/gpu/drm/virtio/virtgpu_drv.h
b/drivers/gpu/drm/virtio/virtgpu_drv.h
index af6ffb696086..589c95822699 100644
--- a/drivers/gpu/drm/virtio/virtgpu_drv.h
+++ b/drivers/gpu/drm/virtio/virtgpu_drv.h
@@ -89,6 +89,7 @@ struct virtio_gpu_object {
uint32_t hw_res_handle;
bool dumb;
bool created;
+ bool detached;
bool host3d_blob, guest_blob;
uint32_t blob_mem, blob_flags;
@@ -277,7 +278,7 @@ struct virtio_gpu_fpriv {
};
/* virtgpu_ioctl.c */
-#define DRM_VIRTIO_NUM_IOCTLS 12
+#define DRM_VIRTIO_NUM_IOCTLS 13
extern struct drm_ioctl_desc virtio_gpu_ioctls[DRM_VIRTIO_NUM_IOCTLS];
void virtio_gpu_create_context(struct drm_device *dev, struct drm_file *file);
@@ -313,6 +314,12 @@ void virtio_gpu_array_put_free(struct
virtio_gpu_object_array *objs);
void virtio_gpu_array_put_free_delayed(struct virtio_gpu_device *vgdev,
struct virtio_gpu_object_array *objs);
void virtio_gpu_array_put_free_work(struct work_struct *work);
+int virtio_gpu_array_prepare(struct virtio_gpu_device *vgdev,
+struct virtio_gpu_object_array *objs);
+int virtio_gpu_gem_host_mem_release(struct virtio_gpu_object *bo);
+int virtio_gpu_gem_madvise(struct virtio_gpu_object *obj, int madv);
+int virtio_gpu_gem_pin(struct virtio_gpu_object *bo);
+void virtio_gpu_gem_unpin(struct virtio_gpu_object *bo);
/* virtgpu_vq.c */
int virtio_gpu_alloc_vbufs(struct virtio_gpu_device *vgdev);
@@ -324,6 +331,8 @@ void virtio_gpu_cmd_create_resource(struct
virtio_gpu_device *vgdev,
struct virtio_gpu_fence *fence);
void virtio_gpu_cmd_unref_resource(struct virtio_gpu_device *vgdev,
struct virtio_gpu_object *bo);
+int virtio_gpu_cmd_release_resource(struct virtio_gpu_device *vgdev,
+ struct virtio_gpu_object *bo);
void virtio_gpu_cmd_transfer_to_host_2d(struct virtio_gpu_device *vgdev,
uint64_t offset,
uint32_t width, uint32_t height,
@@ -344,6 +353,9 @@ void virtio_gpu_object_attach(struct virtio_gpu_device
*vgdev,
struct virtio_gpu_object *obj,
struct virtio_gpu_mem_entry *ents,
unsigned int nents);
+void virtio_gpu_object_detach(struct virtio_gpu_device *vgdev,
+ struct virtio_gpu_object *obj,
+ struct virtio_gpu_fence *fence);
int virtio_gpu_attach_status_page(struct virtio_gpu_device *vgdev);
int virtio_gpu_detach_status_page(struct virtio_gpu_device *vgdev);
void virtio_gpu_cursor_ping(struct virtio_gpu_device *vgdev,
@@ -456,6 +468,8 @@ int virtio_gpu_object_create(struct virtio_gpu_device
*vgdev,
bool virtio_gpu_is_shmem(struct virtio_gpu_object *bo);
+int virtio_gpu_reattach_shmem_object(struct virtio_gpu_object *bo);
+
int virtio_gpu_resource_id_get(struct virtio_gpu_device *vgdev,
uint32_t *resid);
/* virtgpu_prime.c */
@@ -486,4 +500,8 @@ void virtio_gpu_vram_unmap_dma_buf(struct device *dev,
struct sg_table *sgt,
enum dma_data_direction dir);
+/* virtgpu_gem_shrinker.c */
+int virtio_gpu_gem_shrinker_init(struct virtio_gpu_device *vgdev);
+void virtio_gpu_gem_shrinker_fini(struct virtio_gpu_device *vgdev);
+
#endif
diff --git a/drivers/gpu/drm/virtio/virtgpu_gem.c
b/drivers/gpu/drm/virtio/virtgpu_gem.c
index 7db48d17ee3a..0f6480b60b68 100644
--- a/drivers/gpu/drm/virtio/virtgpu_gem.c
+++ b/drivers/gpu/drm/virtio/virtgpu_gem.c
@@ -294,3 +294,71 @@ void virtio_gpu_array_put_free_work(struct work_struct
*work)
}
spin_unlock(>obj_free_lock);
}
+
+int virtio_gpu_array_prepare(struct virtio_gpu_device *vgdev,
+struct virtio_gpu_object_array *objs)
+{
+ struct virtio_gpu_object *bo;
+ int ret = 0;
[Intel-gfx] [PATCH v13 08/10] drm/shmem-helper: Export drm_gem_shmem_get_pages_sgt_locked()
Export drm_gem_shmem_get_pages_sgt_locked() that will be used by virtio-gpu
shrinker during GEM swap-in operation done under the held reservation lock.
Signed-off-by: Dmitry Osipenko
---
drivers/gpu/drm/drm_gem_shmem_helper.c | 3 ++-
include/drm/drm_gem_shmem_helper.h | 1 +
2 files changed, 3 insertions(+), 1 deletion(-)
diff --git a/drivers/gpu/drm/drm_gem_shmem_helper.c
b/drivers/gpu/drm/drm_gem_shmem_helper.c
index 9e94652a141c..dd333610d175 100644
--- a/drivers/gpu/drm/drm_gem_shmem_helper.c
+++ b/drivers/gpu/drm/drm_gem_shmem_helper.c
@@ -849,7 +849,7 @@ struct sg_table *drm_gem_shmem_get_sg_table(struct
drm_gem_shmem_object *shmem)
}
EXPORT_SYMBOL_GPL(drm_gem_shmem_get_sg_table);
-static struct sg_table *drm_gem_shmem_get_pages_sgt_locked(struct
drm_gem_shmem_object *shmem)
+struct sg_table *drm_gem_shmem_get_pages_sgt_locked(struct
drm_gem_shmem_object *shmem)
{
struct drm_gem_object *obj = >base;
int ret;
@@ -887,6 +887,7 @@ static struct sg_table
*drm_gem_shmem_get_pages_sgt_locked(struct drm_gem_shmem_
drm_gem_shmem_put_pages(shmem);
return ERR_PTR(ret);
}
+EXPORT_SYMBOL_GPL(drm_gem_shmem_get_pages_sgt_locked);
/**
* drm_gem_shmem_get_pages_sgt - Pin pages, dma map them, and return a
diff --git a/include/drm/drm_gem_shmem_helper.h
b/include/drm/drm_gem_shmem_helper.h
index e99f1715514b..61aaacc6cb99 100644
--- a/include/drm/drm_gem_shmem_helper.h
+++ b/include/drm/drm_gem_shmem_helper.h
@@ -144,6 +144,7 @@ void drm_gem_shmem_purge(struct drm_gem_shmem_object
*shmem);
struct sg_table *drm_gem_shmem_get_sg_table(struct drm_gem_shmem_object
*shmem);
struct sg_table *drm_gem_shmem_get_pages_sgt(struct drm_gem_shmem_object
*shmem);
+struct sg_table *drm_gem_shmem_get_pages_sgt_locked(struct
drm_gem_shmem_object *shmem);
void drm_gem_shmem_print_info(const struct drm_gem_shmem_object *shmem,
struct drm_printer *p, unsigned int indent);
--
2.39.2
[Intel-gfx] [PATCH v13 07/10] drm/shmem-helper: Remove obsoleted is_iomem test
Everything that uses the mapped buffer should by agnostic to is_iomem.
The only reason for the is_iomem test is is that we're setting shmem->vaddr
to the returned map->vaddr. Now that the shmem->vaddr code is gone, remove
the obsoleted is_iomem test to clean up the code.
Suggested-by: Thomas Zimmermann
Signed-off-by: Dmitry Osipenko
---
drivers/gpu/drm/drm_gem_shmem_helper.c | 6 --
1 file changed, 6 deletions(-)
diff --git a/drivers/gpu/drm/drm_gem_shmem_helper.c
b/drivers/gpu/drm/drm_gem_shmem_helper.c
index a02377a5131b..9e94652a141c 100644
--- a/drivers/gpu/drm/drm_gem_shmem_helper.c
+++ b/drivers/gpu/drm/drm_gem_shmem_helper.c
@@ -432,12 +432,6 @@ int drm_gem_shmem_vmap(struct drm_gem_shmem_object *shmem,
if (obj->import_attach) {
ret = dma_buf_vmap(obj->import_attach->dmabuf, map);
- if (!ret) {
- if (drm_WARN_ON(obj->dev, map->is_iomem)) {
- dma_buf_vunmap(obj->import_attach->dmabuf, map);
- return -EIO;
- }
- }
} else {
pgprot_t prot = PAGE_KERNEL;
--
2.39.2
[Intel-gfx] [PATCH v13 06/10] drm/shmem-helper: Add memory shrinker
Introduce common drm-shmem shrinker for DRM drivers.
To start using drm-shmem shrinker drivers should do the following:
1. Implement evict() callback of GEM object where driver should check
whether object is purgeable or evictable using drm-shmem helpers and
perform the shrinking action
2. Initialize drm-shmem internals using drmm_gem_shmem_init(drm_device),
which will register drm-shmem shrinker
3. Implement madvise IOCTL that will use drm_gem_shmem_madvise()
Signed-off-by: Daniel Almeida
Signed-off-by: Dmitry Osipenko
---
drivers/gpu/drm/drm_gem_shmem_helper.c| 351 +-
.../gpu/drm/panfrost/panfrost_gem_shrinker.c | 9 +-
include/drm/drm_device.h | 10 +-
include/drm/drm_gem_shmem_helper.h| 52 ++-
4 files changed, 402 insertions(+), 20 deletions(-)
diff --git a/drivers/gpu/drm/drm_gem_shmem_helper.c
b/drivers/gpu/drm/drm_gem_shmem_helper.c
index 48df4e87da26..a02377a5131b 100644
--- a/drivers/gpu/drm/drm_gem_shmem_helper.c
+++ b/drivers/gpu/drm/drm_gem_shmem_helper.c
@@ -20,6 +20,7 @@
#include
#include
#include
+#include
#include
#include
@@ -88,8 +89,6 @@ __drm_gem_shmem_create(struct drm_device *dev, size_t size,
bool private)
if (ret)
goto err_release;
- INIT_LIST_HEAD(>madv_list);
-
if (!private) {
/*
* Our buffers are kept pinned, so allocating them
@@ -128,6 +127,57 @@ struct drm_gem_shmem_object *drm_gem_shmem_create(struct
drm_device *dev, size_t
}
EXPORT_SYMBOL_GPL(drm_gem_shmem_create);
+static void drm_gem_shmem_resv_assert_held(struct drm_gem_shmem_object *shmem)
+{
+ /*
+* Destroying the object is a special case.. drm_gem_shmem_free()
+* calls many things that WARN_ON if the obj lock is not held. But
+* acquiring the obj lock in drm_gem_shmem_free() can cause a locking
+* order inversion between reservation_ww_class_mutex and fs_reclaim.
+*
+* This deadlock is not actually possible, because no one should
+* be already holding the lock when drm_gem_shmem_free() is called.
+* Unfortunately lockdep is not aware of this detail. So when the
+* refcount drops to zero, we pretend it is already locked.
+*/
+ if (kref_read(>base.refcount))
+ dma_resv_assert_held(shmem->base.resv);
+}
+
+static bool drm_gem_shmem_is_evictable(struct drm_gem_shmem_object *shmem)
+{
+ dma_resv_assert_held(shmem->base.resv);
+
+ return (shmem->madv >= 0) && shmem->base.funcs->evict &&
+ shmem->pages_use_count && !shmem->pages_pin_count &&
+ !shmem->base.dma_buf && !shmem->base.import_attach &&
+ shmem->sgt && !shmem->evicted;
+}
+
+static void
+drm_gem_shmem_update_pages_state(struct drm_gem_shmem_object *shmem)
+{
+ struct drm_gem_object *obj = >base;
+ struct drm_gem_shmem *shmem_mm = obj->dev->shmem_mm;
+ struct drm_gem_shmem_shrinker *shmem_shrinker = _mm->shrinker;
+
+ drm_gem_shmem_resv_assert_held(shmem);
+
+ if (!shmem_shrinker || obj->import_attach)
+ return;
+
+ if (shmem->madv < 0)
+ drm_gem_lru_remove(>base);
+ else if (drm_gem_shmem_is_evictable(shmem) ||
drm_gem_shmem_is_purgeable(shmem))
+ drm_gem_lru_move_tail(_shrinker->lru_evictable,
>base);
+ else if (shmem->evicted)
+ drm_gem_lru_move_tail(_shrinker->lru_evicted,
>base);
+ else if (!shmem->pages)
+ drm_gem_lru_remove(>base);
+ else
+ drm_gem_lru_move_tail(_shrinker->lru_pinned,
>base);
+}
+
/**
* drm_gem_shmem_free - Free resources associated with a shmem GEM object
* @shmem: shmem GEM object to free
@@ -142,7 +192,8 @@ void drm_gem_shmem_free(struct drm_gem_shmem_object *shmem)
if (obj->import_attach) {
drm_prime_gem_destroy(obj, shmem->sgt);
} else {
- dma_resv_lock(shmem->base.resv, NULL);
+ /* take out shmem GEM object from the memory shrinker */
+ drm_gem_shmem_madvise(shmem, -1);
drm_WARN_ON(obj->dev, shmem->vmap_use_count);
@@ -152,12 +203,10 @@ void drm_gem_shmem_free(struct drm_gem_shmem_object
*shmem)
sg_free_table(shmem->sgt);
kfree(shmem->sgt);
}
- if (shmem->pages)
+ if (shmem->pages_use_count)
drm_gem_shmem_put_pages(shmem);
drm_WARN_ON(obj->dev, shmem->pages_use_count);
-
- dma_resv_unlock(shmem->base.resv);
}
drm_gem_object_release(obj);
@@ -178,6 +227,11 @@ drm_gem_shmem_acquire_pages(stru
[Intel-gfx] [PATCH v13 05/10] drm/shmem-helper: Factor out unpinning part from drm_gem_shmem_purge()
Factor out pages unpinning code from drm_gem_shmem_purge() into new
drm_gem_shmem_unpin_pages(). This prepares code for addition of memory
shrinker support. The new common function will be used by shrinker for
eviction of shmem pages.
Signed-off-by: Dmitry Osipenko
---
drivers/gpu/drm/drm_gem_shmem_helper.c | 18 --
1 file changed, 12 insertions(+), 6 deletions(-)
diff --git a/drivers/gpu/drm/drm_gem_shmem_helper.c
b/drivers/gpu/drm/drm_gem_shmem_helper.c
index 1fcb7d850cc7..48df4e87da26 100644
--- a/drivers/gpu/drm/drm_gem_shmem_helper.c
+++ b/drivers/gpu/drm/drm_gem_shmem_helper.c
@@ -486,25 +486,29 @@ int drm_gem_shmem_madvise(struct drm_gem_shmem_object
*shmem, int madv)
}
EXPORT_SYMBOL(drm_gem_shmem_madvise);
-void drm_gem_shmem_purge(struct drm_gem_shmem_object *shmem)
+static void drm_gem_shmem_unpin_pages(struct drm_gem_shmem_object *shmem)
{
struct drm_gem_object *obj = >base;
struct drm_device *dev = obj->dev;
dma_resv_assert_held(shmem->base.resv);
- drm_WARN_ON(obj->dev, !drm_gem_shmem_is_purgeable(shmem));
-
dma_unmap_sgtable(dev->dev, shmem->sgt, DMA_BIDIRECTIONAL, 0);
+ drm_gem_shmem_release_pages(shmem);
+ drm_vma_node_unmap(>vma_node, dev->anon_inode->i_mapping);
+
sg_free_table(shmem->sgt);
kfree(shmem->sgt);
shmem->sgt = NULL;
+}
- drm_gem_shmem_put_pages(shmem);
+void drm_gem_shmem_purge(struct drm_gem_shmem_object *shmem)
+{
+ struct drm_gem_object *obj = >base;
- shmem->madv = -1;
+ drm_WARN_ON(obj->dev, !drm_gem_shmem_is_purgeable(shmem));
- drm_vma_node_unmap(>vma_node, dev->anon_inode->i_mapping);
+ drm_gem_shmem_unpin_pages(shmem);
drm_gem_free_mmap_offset(obj);
/* Our goal here is to return as much of the memory as
@@ -515,6 +519,8 @@ void drm_gem_shmem_purge(struct drm_gem_shmem_object *shmem)
shmem_truncate_range(file_inode(obj->filp), 0, (loff_t)-1);
invalidate_mapping_pages(file_inode(obj->filp)->i_mapping, 0,
(loff_t)-1);
+
+ shmem->madv = -1;
}
EXPORT_SYMBOL(drm_gem_shmem_purge);
--
2.39.2
[Intel-gfx] [PATCH v13 04/10] drm/shmem-helper: Switch drm_gem_shmem_vmap/vunmap to use pin/unpin
The vmapped pages shall be pinned in memory. Previously get/put pages were implicitly pinning/unpinning the pages. This will no longer be the case with addition of memory shrinker because pages_use_count>0 won't determine whether pages are pinned anymore, while the new pages_pin_count will do that. Switch the vmap/vunmap to use pin/unpin functions in a preparation of addition of the memory shrinker support. Signed-off-by: Dmitry Osipenko --- drivers/gpu/drm/drm_gem_shmem_helper.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/drivers/gpu/drm/drm_gem_shmem_helper.c b/drivers/gpu/drm/drm_gem_shmem_helper.c index 81d61791f874..1fcb7d850cc7 100644 --- a/drivers/gpu/drm/drm_gem_shmem_helper.c +++ b/drivers/gpu/drm/drm_gem_shmem_helper.c @@ -380,7 +380,7 @@ int drm_gem_shmem_vmap(struct drm_gem_shmem_object *shmem, return 0; } - ret = drm_gem_shmem_get_pages(shmem); + ret = drm_gem_shmem_pin_locked(shmem); if (ret) goto err_zero_use; @@ -403,7 +403,7 @@ int drm_gem_shmem_vmap(struct drm_gem_shmem_object *shmem, err_put_pages: if (!obj->import_attach) - drm_gem_shmem_put_pages(shmem); + drm_gem_shmem_unpin_locked(shmem); err_zero_use: shmem->vmap_use_count = 0; @@ -440,7 +440,7 @@ void drm_gem_shmem_vunmap(struct drm_gem_shmem_object *shmem, return; vunmap(shmem->vaddr); - drm_gem_shmem_put_pages(shmem); + drm_gem_shmem_unpin_locked(shmem); } shmem->vaddr = NULL; -- 2.39.2
[Intel-gfx] [PATCH v13 03/10] drm/shmem-helper: Add pages_pin_count field
And new pages_pin_count field to struct drm_gem_shmem_object that will
determine whether pages are evictable by memory shrinker. The pages will
be evictable only when pages_pin_count=0. This patch prepares code for
addition of the memory shrinker that will utilize the new field.
Signed-off-by: Dmitry Osipenko
---
drivers/gpu/drm/drm_gem_shmem_helper.c | 7 +++
include/drm/drm_gem_shmem_helper.h | 9 +
2 files changed, 16 insertions(+)
diff --git a/drivers/gpu/drm/drm_gem_shmem_helper.c
b/drivers/gpu/drm/drm_gem_shmem_helper.c
index 4da9c9c39b9a..81d61791f874 100644
--- a/drivers/gpu/drm/drm_gem_shmem_helper.c
+++ b/drivers/gpu/drm/drm_gem_shmem_helper.c
@@ -277,6 +277,8 @@ static int drm_gem_shmem_pin_locked(struct
drm_gem_shmem_object *shmem)
drm_WARN_ON(obj->dev, obj->import_attach);
ret = drm_gem_shmem_get_pages(shmem);
+ if (!ret)
+ shmem->pages_pin_count++;
return ret;
}
@@ -289,7 +291,12 @@ static void drm_gem_shmem_unpin_locked(struct
drm_gem_shmem_object *shmem)
drm_WARN_ON(obj->dev, obj->import_attach);
+ if (drm_WARN_ON_ONCE(obj->dev, !shmem->pages_pin_count))
+ return;
+
drm_gem_shmem_put_pages(shmem);
+
+ shmem->pages_pin_count--;
}
/**
diff --git a/include/drm/drm_gem_shmem_helper.h
b/include/drm/drm_gem_shmem_helper.h
index 20ddcd799df9..7d823c9fc480 100644
--- a/include/drm/drm_gem_shmem_helper.h
+++ b/include/drm/drm_gem_shmem_helper.h
@@ -39,6 +39,15 @@ struct drm_gem_shmem_object {
*/
unsigned int pages_use_count;
+ /**
+* @pages_pin_count:
+*
+* Reference count on the pinned pages table.
+* The pages allowed to be evicted by memory shrinker
+* only when the count is zero.
+*/
+ unsigned int pages_pin_count;
+
/**
* @madv: State for madvise
*
--
2.39.2
[Intel-gfx] [PATCH v13 02/10] drm/shmem-helper: Factor out pages alloc/release from drm_gem_shmem_get/put_pages()
Factor out pages allocation from drm_gem_shmem_get_pages() into
drm_gem_shmem_acquire_pages() function and similar for the put_pages()
in a preparation for addition of shrinker support to drm-shmem.
Once shrinker will be added, the pages_use_count>0 will no longer determine
whether pages are pinned because pages could be swapped out by the shrinker
and then pages_use_count will be greater than 0 in this case. We will add
new pages_pin_count in a later patch.
The new common drm_gem_shmem_acquire/release_pages() will be used by
shrinker code for performing the page swapping.
Signed-off-by: Dmitry Osipenko
---
drivers/gpu/drm/drm_gem_shmem_helper.c | 67 +-
1 file changed, 54 insertions(+), 13 deletions(-)
diff --git a/drivers/gpu/drm/drm_gem_shmem_helper.c
b/drivers/gpu/drm/drm_gem_shmem_helper.c
index 8fc2a3277486..4da9c9c39b9a 100644
--- a/drivers/gpu/drm/drm_gem_shmem_helper.c
+++ b/drivers/gpu/drm/drm_gem_shmem_helper.c
@@ -165,19 +165,26 @@ void drm_gem_shmem_free(struct drm_gem_shmem_object
*shmem)
}
EXPORT_SYMBOL_GPL(drm_gem_shmem_free);
-static int drm_gem_shmem_get_pages(struct drm_gem_shmem_object *shmem)
+static int
+drm_gem_shmem_acquire_pages(struct drm_gem_shmem_object *shmem)
{
struct drm_gem_object *obj = >base;
struct page **pages;
- if (shmem->pages_use_count++ > 0)
- return 0;
+ dma_resv_assert_held(shmem->base.resv);
+
+ if (shmem->madv < 0) {
+ drm_WARN_ON(obj->dev, shmem->pages);
+ return -ENOMEM;
+ }
+
+ if (drm_WARN_ON(obj->dev, !shmem->pages_use_count))
+ return -EINVAL;
pages = drm_gem_get_pages(obj);
if (IS_ERR(pages)) {
drm_dbg_kms(obj->dev, "Failed to get pages (%ld)\n",
PTR_ERR(pages));
- shmem->pages_use_count = 0;
return PTR_ERR(pages);
}
@@ -196,6 +203,48 @@ static int drm_gem_shmem_get_pages(struct
drm_gem_shmem_object *shmem)
return 0;
}
+static int drm_gem_shmem_get_pages(struct drm_gem_shmem_object *shmem)
+{
+ int err;
+
+ dma_resv_assert_held(shmem->base.resv);
+
+ if (shmem->madv < 0)
+ return -ENOMEM;
+
+ if (shmem->pages_use_count++ > 0)
+ return 0;
+
+ err = drm_gem_shmem_acquire_pages(shmem);
+ if (err)
+ goto err_zero_use;
+
+ return 0;
+
+err_zero_use:
+ shmem->pages_use_count = 0;
+
+ return err;
+}
+
+static void
+drm_gem_shmem_release_pages(struct drm_gem_shmem_object *shmem)
+{
+ struct drm_gem_object *obj = >base;
+
+ dma_resv_assert_held(shmem->base.resv);
+
+#ifdef CONFIG_X86
+ if (shmem->map_wc)
+ set_pages_array_wb(shmem->pages, obj->size >> PAGE_SHIFT);
+#endif
+
+ drm_gem_put_pages(obj, shmem->pages,
+ shmem->pages_mark_dirty_on_put,
+ shmem->pages_mark_accessed_on_put);
+ shmem->pages = NULL;
+}
+
/*
* drm_gem_shmem_put_pages - Decrease use count on the backing pages for a
shmem GEM object
* @shmem: shmem GEM object
@@ -214,15 +263,7 @@ void drm_gem_shmem_put_pages(struct drm_gem_shmem_object
*shmem)
if (--shmem->pages_use_count > 0)
return;
-#ifdef CONFIG_X86
- if (shmem->map_wc)
- set_pages_array_wb(shmem->pages, obj->size >> PAGE_SHIFT);
-#endif
-
- drm_gem_put_pages(obj, shmem->pages,
- shmem->pages_mark_dirty_on_put,
- shmem->pages_mark_accessed_on_put);
- shmem->pages = NULL;
+ drm_gem_shmem_release_pages(shmem);
}
EXPORT_SYMBOL(drm_gem_shmem_put_pages);
--
2.39.2
[Intel-gfx] [PATCH v13 01/10] drm/shmem-helper: Switch to reservation lock
Replace all drm-shmem locks with a GEM reservation lock. This makes locks
consistent with dma-buf locking convention where importers are responsible
for holding reservation lock for all operations performed over dma-bufs,
preventing deadlock between dma-buf importers and exporters.
Suggested-by: Daniel Vetter
Acked-by: Thomas Zimmermann
Signed-off-by: Dmitry Osipenko
---
drivers/gpu/drm/drm_gem_shmem_helper.c| 217 --
drivers/gpu/drm/lima/lima_gem.c | 8 +-
drivers/gpu/drm/panfrost/panfrost_drv.c | 7 +-
.../gpu/drm/panfrost/panfrost_gem_shrinker.c | 6 +-
drivers/gpu/drm/panfrost/panfrost_mmu.c | 19 +-
include/drm/drm_gem_shmem_helper.h| 14 +-
6 files changed, 120 insertions(+), 151 deletions(-)
diff --git a/drivers/gpu/drm/drm_gem_shmem_helper.c
b/drivers/gpu/drm/drm_gem_shmem_helper.c
index 4ea6507a77e5..8fc2a3277486 100644
--- a/drivers/gpu/drm/drm_gem_shmem_helper.c
+++ b/drivers/gpu/drm/drm_gem_shmem_helper.c
@@ -88,8 +88,6 @@ __drm_gem_shmem_create(struct drm_device *dev, size_t size,
bool private)
if (ret)
goto err_release;
- mutex_init(>pages_lock);
- mutex_init(>vmap_lock);
INIT_LIST_HEAD(>madv_list);
if (!private) {
@@ -141,11 +139,13 @@ void drm_gem_shmem_free(struct drm_gem_shmem_object
*shmem)
{
struct drm_gem_object *obj = >base;
- drm_WARN_ON(obj->dev, shmem->vmap_use_count);
-
if (obj->import_attach) {
drm_prime_gem_destroy(obj, shmem->sgt);
} else {
+ dma_resv_lock(shmem->base.resv, NULL);
+
+ drm_WARN_ON(obj->dev, shmem->vmap_use_count);
+
if (shmem->sgt) {
dma_unmap_sgtable(obj->dev->dev, shmem->sgt,
DMA_BIDIRECTIONAL, 0);
@@ -154,18 +154,18 @@ void drm_gem_shmem_free(struct drm_gem_shmem_object
*shmem)
}
if (shmem->pages)
drm_gem_shmem_put_pages(shmem);
- }
- drm_WARN_ON(obj->dev, shmem->pages_use_count);
+ drm_WARN_ON(obj->dev, shmem->pages_use_count);
+
+ dma_resv_unlock(shmem->base.resv);
+ }
drm_gem_object_release(obj);
- mutex_destroy(>pages_lock);
- mutex_destroy(>vmap_lock);
kfree(shmem);
}
EXPORT_SYMBOL_GPL(drm_gem_shmem_free);
-static int drm_gem_shmem_get_pages_locked(struct drm_gem_shmem_object *shmem)
+static int drm_gem_shmem_get_pages(struct drm_gem_shmem_object *shmem)
{
struct drm_gem_object *obj = >base;
struct page **pages;
@@ -197,35 +197,16 @@ static int drm_gem_shmem_get_pages_locked(struct
drm_gem_shmem_object *shmem)
}
/*
- * drm_gem_shmem_get_pages - Allocate backing pages for a shmem GEM object
+ * drm_gem_shmem_put_pages - Decrease use count on the backing pages for a
shmem GEM object
* @shmem: shmem GEM object
*
- * This function makes sure that backing pages exists for the shmem GEM object
- * and increases the use count.
- *
- * Returns:
- * 0 on success or a negative error code on failure.
+ * This function decreases the use count and puts the backing pages when use
drops to zero.
*/
-int drm_gem_shmem_get_pages(struct drm_gem_shmem_object *shmem)
+void drm_gem_shmem_put_pages(struct drm_gem_shmem_object *shmem)
{
struct drm_gem_object *obj = >base;
- int ret;
- drm_WARN_ON(obj->dev, obj->import_attach);
-
- ret = mutex_lock_interruptible(>pages_lock);
- if (ret)
- return ret;
- ret = drm_gem_shmem_get_pages_locked(shmem);
- mutex_unlock(>pages_lock);
-
- return ret;
-}
-EXPORT_SYMBOL(drm_gem_shmem_get_pages);
-
-static void drm_gem_shmem_put_pages_locked(struct drm_gem_shmem_object *shmem)
-{
- struct drm_gem_object *obj = >base;
+ dma_resv_assert_held(shmem->base.resv);
if (drm_WARN_ON_ONCE(obj->dev, !shmem->pages_use_count))
return;
@@ -243,20 +224,32 @@ static void drm_gem_shmem_put_pages_locked(struct
drm_gem_shmem_object *shmem)
shmem->pages_mark_accessed_on_put);
shmem->pages = NULL;
}
+EXPORT_SYMBOL(drm_gem_shmem_put_pages);
-/*
- * drm_gem_shmem_put_pages - Decrease use count on the backing pages for a
shmem GEM object
- * @shmem: shmem GEM object
- *
- * This function decreases the use count and puts the backing pages when use
drops to zero.
- */
-void drm_gem_shmem_put_pages(struct drm_gem_shmem_object *shmem)
+static int drm_gem_shmem_pin_locked(struct drm_gem_shmem_object *shmem)
{
- mutex_lock(>pages_lock);
- drm_gem_shmem_put_pages_locked(shmem);
- mutex_unlock(>pages_lock);
+ struct drm_gem_object *obj = >base;
+ int ret;
+
+ dma_resv_assert_held(shmem->base.resv);
+
+ drm_WARN_ON(o
[Intel-gfx] [PATCH v13 00/10] Add generic memory shrinker to VirtIO-GPU and Panfrost DRM drivers
This series: 1. Adds common drm-shmem memory shrinker 2. Enables shrinker for VirtIO-GPU driver 3. Switches Panfrost driver to the common shrinker Changelog: v13:- Updated virtio-gpu shrinker patch to use drm_gem_shmem_object_pin() directly instead of drm_gem_pin() and dropped patch that exported drm_gem_pin() functions, like was requested by Thomas Zimmermann in v12. v12:- Fixed the "no previous prototype for function" warning reported by kernel build bot for v11. - Fixed the missing reservation lock reported by Intel CI for VGEM driver. Other drivers using drm-shmem were affected similarly to VGEM. The problem was in the dma-buf attachment code path that led to drm-shmem pinning function which assumed the held reservation lock by drm_gem_pin(). In the past that code path was causing trouble for i915 driver and we've changed the locking scheme for the attachment code path in the dma-buf core to let exporters to handle the locking themselves. After a closer investigation, I realized that my assumption about testing of dma-buf export code path using Panfrost driver was incorrect. Now I created additional local test to exrecise the Panfrost export path. I also reproduced the issue reported by the Intel CI for v10. It's all fixed now by making the drm_gem_shmem_pin() to take the resv lock by itself. - Patches are based on top of drm-tip, CC'd intel-gfx CI for testing. v11:- Rebased on a recent linux-next. Added new patch as a result: drm/shmem-helper: Export drm_gem_shmem_get_pages_sgt_locked() It's needed by the virtio-gpu driver to swap-in/unevict shmem object, previously get_pages_sgt() didn't use locking. - Separated the "Add memory shrinker" patch into smaller parts to ease the reviewing, as was requested by Thomas Zimmermann: drm/shmem-helper: Factor out pages alloc/release from drm_gem_shmem_get/put_pages() drm/shmem-helper: Add pages_pin_count field drm/shmem-helper: Switch drm_gem_shmem_vmap/vunmap to use pin/unpin drm/shmem-helper: Factor out unpinning part from drm_gem_shmem_purge() - Addessed the v10 review comments from Thomas Zimmermann: return errno instead of bool, sort code alphabetically, rename function and etc minor changes. - Added new patch to remove the "map->is_iomem" from drm-shmem, as was suggested by Thomas Zimmermann. - Added acks and r-b's that were given to v10. v10:- Was partially applied to misc-fixes/next. https://lore.kernel.org/dri-devel/6c16f303-81df-7ebe-85e9-51bb40a8b...@collabora.com/T/ Dmitry Osipenko (10): drm/shmem-helper: Switch to reservation lock drm/shmem-helper: Factor out pages alloc/release from drm_gem_shmem_get/put_pages() drm/shmem-helper: Add pages_pin_count field drm/shmem-helper: Switch drm_gem_shmem_vmap/vunmap to use pin/unpin drm/shmem-helper: Factor out unpinning part from drm_gem_shmem_purge() drm/shmem-helper: Add memory shrinker drm/shmem-helper: Remove obsoleted is_iomem test drm/shmem-helper: Export drm_gem_shmem_get_pages_sgt_locked() drm/virtio: Support memory shrinking drm/panfrost: Switch to generic memory shrinker drivers/gpu/drm/drm_gem_shmem_helper.c| 613 ++ drivers/gpu/drm/lima/lima_gem.c | 8 +- drivers/gpu/drm/panfrost/Makefile | 1 - drivers/gpu/drm/panfrost/panfrost_device.h| 4 - drivers/gpu/drm/panfrost/panfrost_drv.c | 34 +- drivers/gpu/drm/panfrost/panfrost_gem.c | 30 +- drivers/gpu/drm/panfrost/panfrost_gem.h | 9 - .../gpu/drm/panfrost/panfrost_gem_shrinker.c | 122 drivers/gpu/drm/panfrost/panfrost_job.c | 18 +- drivers/gpu/drm/panfrost/panfrost_mmu.c | 19 +- drivers/gpu/drm/virtio/virtgpu_drv.h | 20 +- drivers/gpu/drm/virtio/virtgpu_gem.c | 68 ++ drivers/gpu/drm/virtio/virtgpu_ioctl.c| 37 ++ drivers/gpu/drm/virtio/virtgpu_kms.c | 8 + drivers/gpu/drm/virtio/virtgpu_object.c | 137 +++- drivers/gpu/drm/virtio/virtgpu_plane.c| 17 +- drivers/gpu/drm/virtio/virtgpu_vq.c | 40 ++ include/drm/drm_device.h | 10 +- include/drm/drm_gem_shmem_helper.h| 83 ++- include/uapi/drm/virtgpu_drm.h| 14 + 20 files changed, 925 insertions(+), 367 deletions(-) delete mode 100644 drivers/gpu/drm/panfrost/panfrost_gem_shrinker.c -- 2.39.2
Re: [Intel-gfx] [PATCH v12 10/11] drm/virtio: Support memory shrinking
On 3/7/23 21:25, Dmitry Osipenko wrote: >> Not really a problem with this patchset, but having such branches looks >> like a bug in the driver's GEM design. Whatever your GEM object needs or >> does, it should be hidden in the implementation. Why is virtio doing this? > There is another "VRAM" VirtIO-GPU BO type that doesn't implement the > pin/unpin callbacks. Perhaps another option was to add the callbacks. Although, the pin/unpin are optional. So yes, there was no need for the extra branch, good catch. -- Best regards, Dmitry
Re: [Intel-gfx] [PATCH v12 10/11] drm/virtio: Support memory shrinking
On 3/7/23 13:42, Thomas Zimmermann wrote:
> Hi
>
> Am 05.03.23 um 23:10 schrieb Dmitry Osipenko:
> [...]
>> *bo_ptr = bo;
>> diff --git a/drivers/gpu/drm/virtio/virtgpu_plane.c
>> b/drivers/gpu/drm/virtio/virtgpu_plane.c
>> index 4c09e313bebc..3f21512ff153 100644
>> --- a/drivers/gpu/drm/virtio/virtgpu_plane.c
>> +++ b/drivers/gpu/drm/virtio/virtgpu_plane.c
>> @@ -238,20 +238,32 @@ static int virtio_gpu_plane_prepare_fb(struct
>> drm_plane *plane,
>> struct virtio_gpu_device *vgdev = dev->dev_private;
>> struct virtio_gpu_framebuffer *vgfb;
>> struct virtio_gpu_object *bo;
>> + int err;
>> if (!new_state->fb)
>> return 0;
>> vgfb = to_virtio_gpu_framebuffer(new_state->fb);
>> bo = gem_to_virtio_gpu_obj(vgfb->base.obj[0]);
>> - if (!bo || (plane->type == DRM_PLANE_TYPE_PRIMARY &&
>> !bo->guest_blob))
>> +
>> + if (virtio_gpu_is_shmem(bo)) {
>
> Not really a problem with this patchset, but having such branches looks
> like a bug in the driver's GEM design. Whatever your GEM object needs or
> does, it should be hidden in the implementation. Why is virtio doing this?
There is another "VRAM" VirtIO-GPU BO type that doesn't implement the
pin/unpin callbacks. Perhaps another option was to add the callbacks.
>> + err = drm_gem_pin(>base.base);
>
> As the driver uses GEM SHMEM, please call drm_gem_shmem_object_pin()
> directly and remove patch [09/11]. Same goes for the _unpin call below.
>
> The problem with generic pinning interfaces is that there's no way of
> specifying where to pin to BO. The problem is most apparent with TTM,
> where hardware often has multiple locations were buffer can be placed
> (VRAM, GART, system memory). So it's really a detail between the driver
> and the memory manager.
>
> These generic internal GEM interfaces should only be used by DRM core
> and helpers. Drivers should use their memory manager's interface.
I'll switch to use drm_gem_shmem_object_pin() directly in v13, maybe add
virtio_gpu_gem_pin() helper for that. Thanks!
--
Best regards,
Dmitry
[Intel-gfx] [PATCH v12 09/11] drm/gem: Export drm_gem_pin/unpin()
Export drm_gem_un/pin() functions. They will be used by VirtIO-GPU driver
for pinning of an active framebuffer, preventing it from swapping out by
memory shrinker.
Signed-off-by: Dmitry Osipenko
---
drivers/gpu/drm/drm_gem.c | 2 ++
drivers/gpu/drm/drm_internal.h | 2 --
include/drm/drm_gem.h | 3 +++
3 files changed, 5 insertions(+), 2 deletions(-)
diff --git a/drivers/gpu/drm/drm_gem.c b/drivers/gpu/drm/drm_gem.c
index 3da34b121c93..6492c47b7142 100644
--- a/drivers/gpu/drm/drm_gem.c
+++ b/drivers/gpu/drm/drm_gem.c
@@ -1153,12 +1153,14 @@ int drm_gem_pin(struct drm_gem_object *obj)
else
return 0;
}
+EXPORT_SYMBOL(drm_gem_pin);
void drm_gem_unpin(struct drm_gem_object *obj)
{
if (obj->funcs->unpin)
obj->funcs->unpin(obj);
}
+EXPORT_SYMBOL(drm_gem_unpin);
int drm_gem_vmap(struct drm_gem_object *obj, struct iosys_map *map)
{
diff --git a/drivers/gpu/drm/drm_internal.h b/drivers/gpu/drm/drm_internal.h
index d7e023bbb0d5..55d0ee7475f7 100644
--- a/drivers/gpu/drm/drm_internal.h
+++ b/drivers/gpu/drm/drm_internal.h
@@ -173,8 +173,6 @@ void drm_gem_release(struct drm_device *dev, struct
drm_file *file_private);
void drm_gem_print_info(struct drm_printer *p, unsigned int indent,
const struct drm_gem_object *obj);
-int drm_gem_pin(struct drm_gem_object *obj);
-void drm_gem_unpin(struct drm_gem_object *obj);
int drm_gem_vmap(struct drm_gem_object *obj, struct iosys_map *map);
void drm_gem_vunmap(struct drm_gem_object *obj, struct iosys_map *map);
diff --git a/include/drm/drm_gem.h b/include/drm/drm_gem.h
index 7bd8e236..f7703cb66569 100644
--- a/include/drm/drm_gem.h
+++ b/include/drm/drm_gem.h
@@ -493,4 +493,7 @@ unsigned long drm_gem_lru_scan(struct drm_gem_lru *lru,
int drm_gem_evict(struct drm_gem_object *obj);
+int drm_gem_pin(struct drm_gem_object *obj);
+void drm_gem_unpin(struct drm_gem_object *obj);
+
#endif /* __DRM_GEM_H__ */
--
2.39.2
[Intel-gfx] [PATCH v12 11/11] drm/panfrost: Switch to generic memory shrinker
Replace Panfrost's custom memory shrinker with a common drm-shmem
memory shrinker.
Tested-by: Steven Price # Firefly-RK3288
Reviewed-by: Steven Price
Signed-off-by: Dmitry Osipenko
---
drivers/gpu/drm/panfrost/Makefile | 1 -
drivers/gpu/drm/panfrost/panfrost_device.h| 4 -
drivers/gpu/drm/panfrost/panfrost_drv.c | 27 ++--
drivers/gpu/drm/panfrost/panfrost_gem.c | 30 ++--
drivers/gpu/drm/panfrost/panfrost_gem.h | 9 --
.../gpu/drm/panfrost/panfrost_gem_shrinker.c | 129 --
drivers/gpu/drm/panfrost/panfrost_job.c | 18 ++-
include/drm/drm_gem_shmem_helper.h| 7 -
8 files changed, 47 insertions(+), 178 deletions(-)
delete mode 100644 drivers/gpu/drm/panfrost/panfrost_gem_shrinker.c
diff --git a/drivers/gpu/drm/panfrost/Makefile
b/drivers/gpu/drm/panfrost/Makefile
index 7da2b3f02ed9..11622e22cf15 100644
--- a/drivers/gpu/drm/panfrost/Makefile
+++ b/drivers/gpu/drm/panfrost/Makefile
@@ -5,7 +5,6 @@ panfrost-y := \
panfrost_device.o \
panfrost_devfreq.o \
panfrost_gem.o \
- panfrost_gem_shrinker.o \
panfrost_gpu.o \
panfrost_job.o \
panfrost_mmu.o \
diff --git a/drivers/gpu/drm/panfrost/panfrost_device.h
b/drivers/gpu/drm/panfrost/panfrost_device.h
index d9ba68cffb77..28f28bbdbda9 100644
--- a/drivers/gpu/drm/panfrost/panfrost_device.h
+++ b/drivers/gpu/drm/panfrost/panfrost_device.h
@@ -116,10 +116,6 @@ struct panfrost_device {
atomic_t pending;
} reset;
- struct mutex shrinker_lock;
- struct list_head shrinker_list;
- struct shrinker shrinker;
-
struct panfrost_devfreq pfdevfreq;
};
diff --git a/drivers/gpu/drm/panfrost/panfrost_drv.c
b/drivers/gpu/drm/panfrost/panfrost_drv.c
index aa292e4a86eb..e29a2e604257 100644
--- a/drivers/gpu/drm/panfrost/panfrost_drv.c
+++ b/drivers/gpu/drm/panfrost/panfrost_drv.c
@@ -169,7 +169,6 @@ panfrost_lookup_bos(struct drm_device *dev,
break;
}
- atomic_inc(>gpu_usecount);
job->mappings[i] = mapping;
}
@@ -394,7 +393,6 @@ static int panfrost_ioctl_madvise(struct drm_device *dev,
void *data,
{
struct panfrost_file_priv *priv = file_priv->driver_priv;
struct drm_panfrost_madvise *args = data;
- struct panfrost_device *pfdev = dev->dev_private;
struct drm_gem_object *gem_obj;
struct panfrost_gem_object *bo;
int ret = 0;
@@ -407,11 +405,15 @@ static int panfrost_ioctl_madvise(struct drm_device *dev,
void *data,
bo = to_panfrost_bo(gem_obj);
+ if (bo->is_heap) {
+ args->retained = 1;
+ goto out_put_object;
+ }
+
ret = dma_resv_lock_interruptible(bo->base.base.resv, NULL);
if (ret)
goto out_put_object;
- mutex_lock(>shrinker_lock);
mutex_lock(>mappings.lock);
if (args->madv == PANFROST_MADV_DONTNEED) {
struct panfrost_gem_mapping *first;
@@ -437,17 +439,8 @@ static int panfrost_ioctl_madvise(struct drm_device *dev,
void *data,
args->retained = drm_gem_shmem_madvise(>base, args->madv);
- if (args->retained) {
- if (args->madv == PANFROST_MADV_DONTNEED)
- list_move_tail(>base.madv_list,
- >shrinker_list);
- else if (args->madv == PANFROST_MADV_WILLNEED)
- list_del_init(>base.madv_list);
- }
-
out_unlock_mappings:
mutex_unlock(>mappings.lock);
- mutex_unlock(>shrinker_lock);
dma_resv_unlock(bo->base.base.resv);
out_put_object:
drm_gem_object_put(gem_obj);
@@ -579,9 +572,6 @@ static int panfrost_probe(struct platform_device *pdev)
ddev->dev_private = pfdev;
pfdev->ddev = ddev;
- mutex_init(>shrinker_lock);
- INIT_LIST_HEAD(>shrinker_list);
-
err = panfrost_device_init(pfdev);
if (err) {
if (err != -EPROBE_DEFER)
@@ -603,10 +593,14 @@ static int panfrost_probe(struct platform_device *pdev)
if (err < 0)
goto err_out1;
- panfrost_gem_shrinker_init(ddev);
+ err = drmm_gem_shmem_init(ddev);
+ if (err < 0)
+ goto err_out2;
return 0;
+err_out2:
+ drm_dev_unregister(ddev);
err_out1:
pm_runtime_disable(pfdev->dev);
panfrost_device_fini(pfdev);
@@ -622,7 +616,6 @@ static int panfrost_remove(struct platform_device *pdev)
struct drm_device *ddev = pfdev->ddev;
drm_dev_unregister(ddev);
- panfrost_gem_shrinker_cleanup(ddev);
pm_runtime_get_sync(pfdev->dev);
pm_runtime_disable(pfdev->dev);
diff --git a/drivers/gpu/drm/panfrost/panfrost_gem.c
b/drivers/gpu/drm/panfrost/panfrost_gem.c
index 3c812
[Intel-gfx] [PATCH v12 08/11] drm/shmem-helper: Export drm_gem_shmem_get_pages_sgt_locked()
Export drm_gem_shmem_get_pages_sgt_locked() that will be used by virtio-gpu
shrinker during GEM swap-in operation done under the held reservation lock.
Signed-off-by: Dmitry Osipenko
---
drivers/gpu/drm/drm_gem_shmem_helper.c | 3 ++-
include/drm/drm_gem_shmem_helper.h | 1 +
2 files changed, 3 insertions(+), 1 deletion(-)
diff --git a/drivers/gpu/drm/drm_gem_shmem_helper.c
b/drivers/gpu/drm/drm_gem_shmem_helper.c
index b814352aae33..b79e74c72b65 100644
--- a/drivers/gpu/drm/drm_gem_shmem_helper.c
+++ b/drivers/gpu/drm/drm_gem_shmem_helper.c
@@ -849,7 +849,7 @@ struct sg_table *drm_gem_shmem_get_sg_table(struct
drm_gem_shmem_object *shmem)
}
EXPORT_SYMBOL_GPL(drm_gem_shmem_get_sg_table);
-static struct sg_table *drm_gem_shmem_get_pages_sgt_locked(struct
drm_gem_shmem_object *shmem)
+struct sg_table *drm_gem_shmem_get_pages_sgt_locked(struct
drm_gem_shmem_object *shmem)
{
struct drm_gem_object *obj = >base;
int ret;
@@ -887,6 +887,7 @@ static struct sg_table
*drm_gem_shmem_get_pages_sgt_locked(struct drm_gem_shmem_
drm_gem_shmem_put_pages(shmem);
return ERR_PTR(ret);
}
+EXPORT_SYMBOL_GPL(drm_gem_shmem_get_pages_sgt_locked);
/**
* drm_gem_shmem_get_pages_sgt - Pin pages, dma map them, and return a
diff --git a/include/drm/drm_gem_shmem_helper.h
b/include/drm/drm_gem_shmem_helper.h
index e99f1715514b..61aaacc6cb99 100644
--- a/include/drm/drm_gem_shmem_helper.h
+++ b/include/drm/drm_gem_shmem_helper.h
@@ -144,6 +144,7 @@ void drm_gem_shmem_purge(struct drm_gem_shmem_object
*shmem);
struct sg_table *drm_gem_shmem_get_sg_table(struct drm_gem_shmem_object
*shmem);
struct sg_table *drm_gem_shmem_get_pages_sgt(struct drm_gem_shmem_object
*shmem);
+struct sg_table *drm_gem_shmem_get_pages_sgt_locked(struct
drm_gem_shmem_object *shmem);
void drm_gem_shmem_print_info(const struct drm_gem_shmem_object *shmem,
struct drm_printer *p, unsigned int indent);
--
2.39.2
[Intel-gfx] [PATCH v12 10/11] drm/virtio: Support memory shrinking
Support generic drm-shmem memory shrinker and add new madvise IOCTL to
the VirtIO-GPU driver. BO cache manager of Mesa driver will mark BOs as
"don't need" using the new IOCTL to let shrinker purge the marked BOs on
OOM, the shrinker will also evict unpurgeable shmem BOs from memory if
guest supports SWAP file or partition.
Acked-by: Gerd Hoffmann
Signed-off-by: Daniel Almeida
Signed-off-by: Dmitry Osipenko
---
drivers/gpu/drm/virtio/virtgpu_drv.h| 18 +++-
drivers/gpu/drm/virtio/virtgpu_gem.c| 52 +
drivers/gpu/drm/virtio/virtgpu_ioctl.c | 37 +++
drivers/gpu/drm/virtio/virtgpu_kms.c| 8 ++
drivers/gpu/drm/virtio/virtgpu_object.c | 137 +++-
drivers/gpu/drm/virtio/virtgpu_plane.c | 22 +++-
drivers/gpu/drm/virtio/virtgpu_vq.c | 40 +++
include/uapi/drm/virtgpu_drm.h | 14 +++
8 files changed, 297 insertions(+), 31 deletions(-)
diff --git a/drivers/gpu/drm/virtio/virtgpu_drv.h
b/drivers/gpu/drm/virtio/virtgpu_drv.h
index af6ffb696086..07eb8d3e5cfd 100644
--- a/drivers/gpu/drm/virtio/virtgpu_drv.h
+++ b/drivers/gpu/drm/virtio/virtgpu_drv.h
@@ -89,6 +89,7 @@ struct virtio_gpu_object {
uint32_t hw_res_handle;
bool dumb;
bool created;
+ bool detached;
bool host3d_blob, guest_blob;
uint32_t blob_mem, blob_flags;
@@ -277,7 +278,7 @@ struct virtio_gpu_fpriv {
};
/* virtgpu_ioctl.c */
-#define DRM_VIRTIO_NUM_IOCTLS 12
+#define DRM_VIRTIO_NUM_IOCTLS 13
extern struct drm_ioctl_desc virtio_gpu_ioctls[DRM_VIRTIO_NUM_IOCTLS];
void virtio_gpu_create_context(struct drm_device *dev, struct drm_file *file);
@@ -313,6 +314,10 @@ void virtio_gpu_array_put_free(struct
virtio_gpu_object_array *objs);
void virtio_gpu_array_put_free_delayed(struct virtio_gpu_device *vgdev,
struct virtio_gpu_object_array *objs);
void virtio_gpu_array_put_free_work(struct work_struct *work);
+int virtio_gpu_array_prepare(struct virtio_gpu_device *vgdev,
+struct virtio_gpu_object_array *objs);
+int virtio_gpu_gem_host_mem_release(struct virtio_gpu_object *bo);
+int virtio_gpu_gem_madvise(struct virtio_gpu_object *obj, int madv);
/* virtgpu_vq.c */
int virtio_gpu_alloc_vbufs(struct virtio_gpu_device *vgdev);
@@ -324,6 +329,8 @@ void virtio_gpu_cmd_create_resource(struct
virtio_gpu_device *vgdev,
struct virtio_gpu_fence *fence);
void virtio_gpu_cmd_unref_resource(struct virtio_gpu_device *vgdev,
struct virtio_gpu_object *bo);
+int virtio_gpu_cmd_release_resource(struct virtio_gpu_device *vgdev,
+ struct virtio_gpu_object *bo);
void virtio_gpu_cmd_transfer_to_host_2d(struct virtio_gpu_device *vgdev,
uint64_t offset,
uint32_t width, uint32_t height,
@@ -344,6 +351,9 @@ void virtio_gpu_object_attach(struct virtio_gpu_device
*vgdev,
struct virtio_gpu_object *obj,
struct virtio_gpu_mem_entry *ents,
unsigned int nents);
+void virtio_gpu_object_detach(struct virtio_gpu_device *vgdev,
+ struct virtio_gpu_object *obj,
+ struct virtio_gpu_fence *fence);
int virtio_gpu_attach_status_page(struct virtio_gpu_device *vgdev);
int virtio_gpu_detach_status_page(struct virtio_gpu_device *vgdev);
void virtio_gpu_cursor_ping(struct virtio_gpu_device *vgdev,
@@ -456,6 +466,8 @@ int virtio_gpu_object_create(struct virtio_gpu_device
*vgdev,
bool virtio_gpu_is_shmem(struct virtio_gpu_object *bo);
+int virtio_gpu_reattach_shmem_object(struct virtio_gpu_object *bo);
+
int virtio_gpu_resource_id_get(struct virtio_gpu_device *vgdev,
uint32_t *resid);
/* virtgpu_prime.c */
@@ -486,4 +498,8 @@ void virtio_gpu_vram_unmap_dma_buf(struct device *dev,
struct sg_table *sgt,
enum dma_data_direction dir);
+/* virtgpu_gem_shrinker.c */
+int virtio_gpu_gem_shrinker_init(struct virtio_gpu_device *vgdev);
+void virtio_gpu_gem_shrinker_fini(struct virtio_gpu_device *vgdev);
+
#endif
diff --git a/drivers/gpu/drm/virtio/virtgpu_gem.c
b/drivers/gpu/drm/virtio/virtgpu_gem.c
index 7db48d17ee3a..8f65911b1e99 100644
--- a/drivers/gpu/drm/virtio/virtgpu_gem.c
+++ b/drivers/gpu/drm/virtio/virtgpu_gem.c
@@ -294,3 +294,55 @@ void virtio_gpu_array_put_free_work(struct work_struct
*work)
}
spin_unlock(>obj_free_lock);
}
+
+int virtio_gpu_array_prepare(struct virtio_gpu_device *vgdev,
+struct virtio_gpu_object_array *objs)
+{
+ struct virtio_gpu_object *bo;
+ int ret = 0;
+ u32 i;
+
+ for (i = 0; i < objs->nents; i++) {
+ bo = gem_to_virtio_g
[Intel-gfx] [PATCH v12 07/11] drm/shmem-helper: Remove obsoleted is_iomem test
Everything that uses the mapped buffer should by agnostic to is_iomem.
The only reason for the is_iomem test is is that we're setting shmem->vaddr
to the returned map->vaddr. Now that the shmem->vaddr code is gone, remove
the obsoleted is_iomem test to clean up the code.
Suggested-by: Thomas Zimmermann
Signed-off-by: Dmitry Osipenko
---
drivers/gpu/drm/drm_gem_shmem_helper.c | 6 --
1 file changed, 6 deletions(-)
diff --git a/drivers/gpu/drm/drm_gem_shmem_helper.c
b/drivers/gpu/drm/drm_gem_shmem_helper.c
index e47cc45f39a1..b814352aae33 100644
--- a/drivers/gpu/drm/drm_gem_shmem_helper.c
+++ b/drivers/gpu/drm/drm_gem_shmem_helper.c
@@ -432,12 +432,6 @@ int drm_gem_shmem_vmap(struct drm_gem_shmem_object *shmem,
if (obj->import_attach) {
ret = dma_buf_vmap(obj->import_attach->dmabuf, map);
- if (!ret) {
- if (drm_WARN_ON(obj->dev, map->is_iomem)) {
- dma_buf_vunmap(obj->import_attach->dmabuf, map);
- return -EIO;
- }
- }
} else {
pgprot_t prot = PAGE_KERNEL;
--
2.39.2
[Intel-gfx] [PATCH v12 05/11] drm/shmem-helper: Factor out unpinning part from drm_gem_shmem_purge()
Factor out pages unpinning code from drm_gem_shmem_purge() into new
drm_gem_shmem_unpin_pages(). This prepares code for addition of memory
shrinker support. The new common function will be used by shrinker for
eviction of shmem pages.
Signed-off-by: Dmitry Osipenko
---
drivers/gpu/drm/drm_gem_shmem_helper.c | 18 --
1 file changed, 12 insertions(+), 6 deletions(-)
diff --git a/drivers/gpu/drm/drm_gem_shmem_helper.c
b/drivers/gpu/drm/drm_gem_shmem_helper.c
index e2324e857847..474918383b3e 100644
--- a/drivers/gpu/drm/drm_gem_shmem_helper.c
+++ b/drivers/gpu/drm/drm_gem_shmem_helper.c
@@ -486,25 +486,29 @@ int drm_gem_shmem_madvise(struct drm_gem_shmem_object
*shmem, int madv)
}
EXPORT_SYMBOL(drm_gem_shmem_madvise);
-void drm_gem_shmem_purge(struct drm_gem_shmem_object *shmem)
+static void drm_gem_shmem_unpin_pages(struct drm_gem_shmem_object *shmem)
{
struct drm_gem_object *obj = >base;
struct drm_device *dev = obj->dev;
dma_resv_assert_held(shmem->base.resv);
- drm_WARN_ON(obj->dev, !drm_gem_shmem_is_purgeable(shmem));
-
dma_unmap_sgtable(dev->dev, shmem->sgt, DMA_BIDIRECTIONAL, 0);
+ drm_gem_shmem_release_pages(shmem);
+ drm_vma_node_unmap(>vma_node, dev->anon_inode->i_mapping);
+
sg_free_table(shmem->sgt);
kfree(shmem->sgt);
shmem->sgt = NULL;
+}
- drm_gem_shmem_put_pages(shmem);
+void drm_gem_shmem_purge(struct drm_gem_shmem_object *shmem)
+{
+ struct drm_gem_object *obj = >base;
- shmem->madv = -1;
+ drm_WARN_ON(obj->dev, !drm_gem_shmem_is_purgeable(shmem));
- drm_vma_node_unmap(>vma_node, dev->anon_inode->i_mapping);
+ drm_gem_shmem_unpin_pages(shmem);
drm_gem_free_mmap_offset(obj);
/* Our goal here is to return as much of the memory as
@@ -515,6 +519,8 @@ void drm_gem_shmem_purge(struct drm_gem_shmem_object *shmem)
shmem_truncate_range(file_inode(obj->filp), 0, (loff_t)-1);
invalidate_mapping_pages(file_inode(obj->filp)->i_mapping, 0,
(loff_t)-1);
+
+ shmem->madv = -1;
}
EXPORT_SYMBOL(drm_gem_shmem_purge);
--
2.39.2
[Intel-gfx] [PATCH v12 06/11] drm/shmem-helper: Add memory shrinker
Introduce common drm-shmem shrinker for DRM drivers.
To start using drm-shmem shrinker drivers should do the following:
1. Implement evict() callback of GEM object where driver should check
whether object is purgeable or evictable using drm-shmem helpers and
perform the shrinking action
2. Initialize drm-shmem internals using drmm_gem_shmem_init(drm_device),
which will register drm-shmem shrinker
3. Implement madvise IOCTL that will use drm_gem_shmem_madvise()
Signed-off-by: Daniel Almeida
Signed-off-by: Dmitry Osipenko
---
drivers/gpu/drm/drm_gem_shmem_helper.c| 351 +-
.../gpu/drm/panfrost/panfrost_gem_shrinker.c | 9 +-
include/drm/drm_device.h | 10 +-
include/drm/drm_gem_shmem_helper.h| 52 ++-
4 files changed, 402 insertions(+), 20 deletions(-)
diff --git a/drivers/gpu/drm/drm_gem_shmem_helper.c
b/drivers/gpu/drm/drm_gem_shmem_helper.c
index 474918383b3e..e47cc45f39a1 100644
--- a/drivers/gpu/drm/drm_gem_shmem_helper.c
+++ b/drivers/gpu/drm/drm_gem_shmem_helper.c
@@ -20,6 +20,7 @@
#include
#include
#include
+#include
#include
#include
@@ -88,8 +89,6 @@ __drm_gem_shmem_create(struct drm_device *dev, size_t size,
bool private)
if (ret)
goto err_release;
- INIT_LIST_HEAD(>madv_list);
-
if (!private) {
/*
* Our buffers are kept pinned, so allocating them
@@ -128,6 +127,57 @@ struct drm_gem_shmem_object *drm_gem_shmem_create(struct
drm_device *dev, size_t
}
EXPORT_SYMBOL_GPL(drm_gem_shmem_create);
+static void drm_gem_shmem_resv_assert_held(struct drm_gem_shmem_object *shmem)
+{
+ /*
+* Destroying the object is a special case.. drm_gem_shmem_free()
+* calls many things that WARN_ON if the obj lock is not held. But
+* acquiring the obj lock in drm_gem_shmem_free() can cause a locking
+* order inversion between reservation_ww_class_mutex and fs_reclaim.
+*
+* This deadlock is not actually possible, because no one should
+* be already holding the lock when drm_gem_shmem_free() is called.
+* Unfortunately lockdep is not aware of this detail. So when the
+* refcount drops to zero, we pretend it is already locked.
+*/
+ if (kref_read(>base.refcount))
+ dma_resv_assert_held(shmem->base.resv);
+}
+
+static bool drm_gem_shmem_is_evictable(struct drm_gem_shmem_object *shmem)
+{
+ dma_resv_assert_held(shmem->base.resv);
+
+ return (shmem->madv >= 0) && shmem->base.funcs->evict &&
+ shmem->pages_use_count && !shmem->pages_pin_count &&
+ !shmem->base.dma_buf && !shmem->base.import_attach &&
+ shmem->sgt && !shmem->evicted;
+}
+
+static void
+drm_gem_shmem_update_pages_state(struct drm_gem_shmem_object *shmem)
+{
+ struct drm_gem_object *obj = >base;
+ struct drm_gem_shmem *shmem_mm = obj->dev->shmem_mm;
+ struct drm_gem_shmem_shrinker *shmem_shrinker = _mm->shrinker;
+
+ drm_gem_shmem_resv_assert_held(shmem);
+
+ if (!shmem_shrinker || obj->import_attach)
+ return;
+
+ if (shmem->madv < 0)
+ drm_gem_lru_remove(>base);
+ else if (drm_gem_shmem_is_evictable(shmem) ||
drm_gem_shmem_is_purgeable(shmem))
+ drm_gem_lru_move_tail(_shrinker->lru_evictable,
>base);
+ else if (shmem->evicted)
+ drm_gem_lru_move_tail(_shrinker->lru_evicted,
>base);
+ else if (!shmem->pages)
+ drm_gem_lru_remove(>base);
+ else
+ drm_gem_lru_move_tail(_shrinker->lru_pinned,
>base);
+}
+
/**
* drm_gem_shmem_free - Free resources associated with a shmem GEM object
* @shmem: shmem GEM object to free
@@ -142,7 +192,8 @@ void drm_gem_shmem_free(struct drm_gem_shmem_object *shmem)
if (obj->import_attach) {
drm_prime_gem_destroy(obj, shmem->sgt);
} else {
- dma_resv_lock(shmem->base.resv, NULL);
+ /* take out shmem GEM object from the memory shrinker */
+ drm_gem_shmem_madvise(shmem, -1);
drm_WARN_ON(obj->dev, shmem->vmap_use_count);
@@ -152,12 +203,10 @@ void drm_gem_shmem_free(struct drm_gem_shmem_object
*shmem)
sg_free_table(shmem->sgt);
kfree(shmem->sgt);
}
- if (shmem->pages)
+ if (shmem->pages_use_count)
drm_gem_shmem_put_pages(shmem);
drm_WARN_ON(obj->dev, shmem->pages_use_count);
-
- dma_resv_unlock(shmem->base.resv);
}
drm_gem_object_release(obj);
@@ -178,6 +227,11 @@ drm_gem_shmem_acquire_pages(stru
[Intel-gfx] [PATCH v12 03/11] drm/shmem-helper: Add pages_pin_count field
And new pages_pin_count field to struct drm_gem_shmem_object that will
determine whether pages are evictable by memory shrinker. The pages will
be evictable only when pages_pin_count=0. This patch prepares code for
addition of the memory shrinker that will utilize the new field.
Signed-off-by: Dmitry Osipenko
---
drivers/gpu/drm/drm_gem_shmem_helper.c | 7 +++
include/drm/drm_gem_shmem_helper.h | 9 +
2 files changed, 16 insertions(+)
diff --git a/drivers/gpu/drm/drm_gem_shmem_helper.c
b/drivers/gpu/drm/drm_gem_shmem_helper.c
index a62c41336a7f..0c3abb0c2ea2 100644
--- a/drivers/gpu/drm/drm_gem_shmem_helper.c
+++ b/drivers/gpu/drm/drm_gem_shmem_helper.c
@@ -277,6 +277,8 @@ static int drm_gem_shmem_pin_locked(struct
drm_gem_shmem_object *shmem)
drm_WARN_ON(obj->dev, obj->import_attach);
ret = drm_gem_shmem_get_pages(shmem);
+ if (!ret)
+ shmem->pages_pin_count++;
return ret;
}
@@ -289,7 +291,12 @@ static void drm_gem_shmem_unpin_locked(struct
drm_gem_shmem_object *shmem)
drm_WARN_ON(obj->dev, obj->import_attach);
+ if (drm_WARN_ON_ONCE(obj->dev, !shmem->pages_pin_count))
+ return;
+
drm_gem_shmem_put_pages(shmem);
+
+ shmem->pages_pin_count--;
}
/**
diff --git a/include/drm/drm_gem_shmem_helper.h
b/include/drm/drm_gem_shmem_helper.h
index 20ddcd799df9..7d823c9fc480 100644
--- a/include/drm/drm_gem_shmem_helper.h
+++ b/include/drm/drm_gem_shmem_helper.h
@@ -39,6 +39,15 @@ struct drm_gem_shmem_object {
*/
unsigned int pages_use_count;
+ /**
+* @pages_pin_count:
+*
+* Reference count on the pinned pages table.
+* The pages allowed to be evicted by memory shrinker
+* only when the count is zero.
+*/
+ unsigned int pages_pin_count;
+
/**
* @madv: State for madvise
*
--
2.39.2
[Intel-gfx] [PATCH v12 04/11] drm/shmem-helper: Switch drm_gem_shmem_vmap/vunmap to use pin/unpin
The vmapped pages shall be pinned in memory. Previously get/put pages were implicitly pinning/unpinning the pages. This will no longer be the case with addition of memory shrinker because pages_use_count>0 won't determine whether pages are pinned anymore, while the new pages_pin_count will do that. Switch the vmap/vunmap to use pin/unpin functions in a preparation of addition of the memory shrinker support. Signed-off-by: Dmitry Osipenko --- drivers/gpu/drm/drm_gem_shmem_helper.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/drivers/gpu/drm/drm_gem_shmem_helper.c b/drivers/gpu/drm/drm_gem_shmem_helper.c index 0c3abb0c2ea2..e2324e857847 100644 --- a/drivers/gpu/drm/drm_gem_shmem_helper.c +++ b/drivers/gpu/drm/drm_gem_shmem_helper.c @@ -380,7 +380,7 @@ int drm_gem_shmem_vmap(struct drm_gem_shmem_object *shmem, return 0; } - ret = drm_gem_shmem_get_pages(shmem); + ret = drm_gem_shmem_pin_locked(shmem); if (ret) goto err_zero_use; @@ -403,7 +403,7 @@ int drm_gem_shmem_vmap(struct drm_gem_shmem_object *shmem, err_put_pages: if (!obj->import_attach) - drm_gem_shmem_put_pages(shmem); + drm_gem_shmem_unpin_locked(shmem); err_zero_use: shmem->vmap_use_count = 0; @@ -440,7 +440,7 @@ void drm_gem_shmem_vunmap(struct drm_gem_shmem_object *shmem, return; vunmap(shmem->vaddr); - drm_gem_shmem_put_pages(shmem); + drm_gem_shmem_unpin_locked(shmem); } shmem->vaddr = NULL; -- 2.39.2
