[BUG] Circular locking dependency - DRM/CMA/MM/hotplug/...
On Fri, Mar 21 2014, Laura Abbott wrote:
> From: Laura Abbott
> Date: Tue, 25 Feb 2014 11:01:19 -0800
> Subject: [PATCH] cma: Remove potential deadlock situation
>
> CMA locking is currently very coarse. The cma_mutex protects both
> the bitmap and avoids concurrency with alloc_contig_range. There
> are several situations which may result in a deadlock on the CMA
> mutex currently, mostly involving AB/BA situations with alloc and
> free. Fix this issue by protecting the bitmap with a mutex per CMA
> region and use the existing mutex for protecting against concurrency
> with alloc_contig_range.
>
> Signed-off-by: Laura Abbott
Acked-by: Michal Nazarewicz
Furthermore, since CMA regions are always MAX_ORDER-page or pageblock
(whichever is bigger) aligned, we could use two mutexes per CMA region:
one protecting the bitmap and the other one protecting calls to
alloc_contig_range touching given region.
On the other hand, we could also go the other way and have two global
mutexes: one protecting all the bitmaps in all the regions and another
protecting calls to alloc_contig_range.
Either way, I think the below patch should work and fix the problem.
> ---
> drivers/base/dma-contiguous.c | 32 +---
> 1 file changed, 25 insertions(+), 7 deletions(-)
>
> diff --git a/drivers/base/dma-contiguous.c b/drivers/base/dma-contiguous.c
> index 165c2c2..dfb48ef 100644
> --- a/drivers/base/dma-contiguous.c
> +++ b/drivers/base/dma-contiguous.c
> @@ -37,6 +37,7 @@ struct cma {
> unsigned long base_pfn;
> unsigned long count;
> unsigned long *bitmap;
> + struct mutexlock;
> };
>
> struct cma *dma_contiguous_default_area;
> @@ -161,6 +162,7 @@ static int __init cma_activate_area(struct cma *cma)
> init_cma_reserved_pageblock(pfn_to_page(base_pfn));
> } while (--i);
>
> + mutex_init(>lock);
> return 0;
> }
>
> @@ -261,6 +263,13 @@ err:
> return ret;
> }
>
> +static void clear_cma_bitmap(struct cma *cma, unsigned long pfn, int count)
> +{
> + mutex_lock(>lock);
> + bitmap_clear(cma->bitmap, pfn - cma->base_pfn, count);
> + mutex_unlock(>lock);
> +}
> +
> /**
> * dma_alloc_from_contiguous() - allocate pages from contiguous area
> * @dev: Pointer to device for which the allocation is performed.
> @@ -294,30 +303,41 @@ struct page *dma_alloc_from_contiguous(struct device
> *dev, int count,
>
> mask = (1 << align) - 1;
>
> - mutex_lock(_mutex);
>
> for (;;) {
> + mutex_lock(>lock);
> pageno = bitmap_find_next_zero_area(cma->bitmap, cma->count,
> start, count, mask);
> - if (pageno >= cma->count)
> + if (pageno >= cma->count) {
> + mutex_unlock(_mutex);
> break;
> + }
> + bitmap_set(cma->bitmap, pageno, count);
> + /*
> + * It's safe to drop the lock here. We've marked this region for
> + * our exclusive use. If the migration fails we will take the
> + * lock again and unmark it.
> + */
> + mutex_unlock(>lock);
>
> pfn = cma->base_pfn + pageno;
> + mutex_lock(_mutex);
> ret = alloc_contig_range(pfn, pfn + count, MIGRATE_CMA);
> + mutex_unlock(_mutex);
> if (ret == 0) {
> - bitmap_set(cma->bitmap, pageno, count);
> page = pfn_to_page(pfn);
> break;
> } else if (ret != -EBUSY) {
> + clear_cma_bitmap(cma, pfn, count);
> break;
> }
> + clear_cma_bitmap(cma, pfn, count);
> pr_debug("%s(): memory range at %p is busy, retrying\n",
>__func__, pfn_to_page(pfn));
> /* try again with a bit different memory target */
> start = pageno + mask + 1;
> }
>
> - mutex_unlock(_mutex);
> pr_debug("%s(): returned %p\n", __func__, page);
> return page;
> }
> @@ -350,10 +370,8 @@ bool dma_release_from_contiguous(struct device *dev,
> struct page *pages,
>
> VM_BUG_ON(pfn + count > cma->base_pfn + cma->count);
>
> - mutex_lock(_mutex);
> - bitmap_clear(cma->bitmap, pfn - cma->base_pfn, count);
> free_contig_range(pfn, count);
> - mutex_unlock(_mutex);
> + clear_cma_bitmap(cma, pfn, count);
>
> return true;
> }
> --
> Code Aurora Forum chooses to take this file under the GPL v 2 license only.
> --
> Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
> hosted by The Linux Foundation
--
Best regards, _ _
.o. | Liege of Serenely Enlightened Majesty of o' \,=./ `o
..o | Computer Science, Micha? ?mina86? Nazarewicz(o o)
ooo +--ooO--(_)--Ooo--
--
[BUG] Circular locking dependency - DRM/CMA/MM/hotplug/...
On 2/18/2014 9:44 AM, Michal Nazarewicz wrote:
>> On 2014-02-12 17:33, Russell King - ARM Linux wrote:
>>> What if we did these changes:
>>>
>>> struct page *dma_alloc_from_contiguous(struct device *dev, int count,
>>> unsigned int align)
>>> {
>>> ...
>>> mutex_lock(_mutex);
>>> ...
>>> for (;;) {
>>> pageno = bitmap_find_next_zero_area(cma->bitmap,
>>> cma->count,
>>> start, count, mask);
>>> if (pageno >= cma->count)
>>> break;
>>>
>>> pfn = cma->base_pfn + pageno;
>>> + bitmap_set(cma->bitmap, pageno, count);
>>> + mutex_unlock(_mutex);
>>> ret = alloc_contig_range(pfn, pfn + count, MIGRATE_CMA);
>>> + mutex_lock(_mutex);
>>> if (ret == 0) {
>>> - bitmap_set(cma->bitmap, pageno, count);
>>> page = pfn_to_page(pfn);
>>> break;
>>> - } else if (ret != -EBUSY) {
>>> + }
>>> + bitmap_clear(cma->bitmap, pageno, count);
>>> + if (ret != -EBUSY) {
>>> break;
>>> }
>>> ...
>>> mutex_unlock(_mutex);
>>> pr_debug("%s(): returned %p\n", __func__, page);
>>> return page;
>>> }
>
> Like Marek said, this will fail if two concurrent calls to
> alloc_contig_range are made such that they operate on the same pageblock
> (which is possible as the allocated regions do not need to be pageblock
> aligned).
>
> Another mutex could be added just for this one call, but as I understand
> this would not solve the problem.
>
>>> bool dma_release_from_contiguous(struct device *dev, struct page *pages,
>>> int count)
>>> {
>>> ...
>>> + free_contig_range(pfn, count);
>>> mutex_lock(_mutex);
>>> bitmap_clear(cma->bitmap, pfn - cma->base_pfn, count);
>>> - free_contig_range(pfn, count);
>>> mutex_unlock(_mutex);
>>> ...
>>> }
>
> This *should* be fine. Didn't test it.
>
I managed to hit a different deadlock that had a similar root cause.
I also managed to independently come up with a similar solution. This
has been tested somewhat but not in wide distribution.
Thanks,
Laura
- 8< --
>From 2aa000fbd4189d967c45c4f1ac5aee812ed83082 Mon Sep 17 00:00:00 2001
From: Laura Abbott
Date: Tue, 25 Feb 2014 11:01:19 -0800
Subject: [PATCH] cma: Remove potential deadlock situation
CMA locking is currently very coarse. The cma_mutex protects both
the bitmap and avoids concurrency with alloc_contig_range. There
are several situations which may result in a deadlock on the CMA
mutex currently, mostly involving AB/BA situations with alloc and
free. Fix this issue by protecting the bitmap with a mutex per CMA
region and use the existing mutex for protecting against concurrency
with alloc_contig_range.
Signed-off-by: Laura Abbott
---
drivers/base/dma-contiguous.c | 32 +---
1 file changed, 25 insertions(+), 7 deletions(-)
diff --git a/drivers/base/dma-contiguous.c b/drivers/base/dma-contiguous.c
index 165c2c2..dfb48ef 100644
--- a/drivers/base/dma-contiguous.c
+++ b/drivers/base/dma-contiguous.c
@@ -37,6 +37,7 @@ struct cma {
unsigned long base_pfn;
unsigned long count;
unsigned long *bitmap;
+ struct mutexlock;
};
struct cma *dma_contiguous_default_area;
@@ -161,6 +162,7 @@ static int __init cma_activate_area(struct cma *cma)
init_cma_reserved_pageblock(pfn_to_page(base_pfn));
} while (--i);
+ mutex_init(>lock);
return 0;
}
@@ -261,6 +263,13 @@ err:
return ret;
}
+static void clear_cma_bitmap(struct cma *cma, unsigned long pfn, int count)
+{
+ mutex_lock(>lock);
+ bitmap_clear(cma->bitmap, pfn - cma->base_pfn, count);
+ mutex_unlock(>lock);
+}
+
/**
* dma_alloc_from_contiguous() - allocate pages from contiguous area
* @dev: Pointer to device for which the allocation is performed.
@@ -294,30 +303,41 @@ struct page *dma_alloc_from_contiguous(struct device
*dev, int count,
mask = (1 << align) - 1;
- mutex_lock(_mutex);
for (;;) {
+ mutex_lock(>lock);
pageno = bitmap_find_next_zero_area(cma->bitmap, cma->count,
start, count, mask);
- if (pageno >= cma->count)
+ if (pageno >= cma->count) {
+ mutex_unlock(_mutex);
break;
+ }
+ bitmap_set(cma->bitmap, pageno, count);
+ /*
+* It's safe to drop the lock here. We've marked this region for
+* our exclusive use. If the migration fails we will take the
+
[BUG] Circular locking dependency - DRM/CMA/MM/hotplug/...
> On 2014-02-12 17:33, Russell King - ARM Linux wrote:
>> What if we did these changes:
>>
>> struct page *dma_alloc_from_contiguous(struct device *dev, int count,
>> unsigned int align)
>> {
>> ...
>> mutex_lock(_mutex);
>> ...
>> for (;;) {
>> pageno = bitmap_find_next_zero_area(cma->bitmap, cma->count,
>> start, count, mask);
>> if (pageno >= cma->count)
>> break;
>>
>> pfn = cma->base_pfn + pageno;
>> + bitmap_set(cma->bitmap, pageno, count);
>> + mutex_unlock(_mutex);
>> ret = alloc_contig_range(pfn, pfn + count, MIGRATE_CMA);
>> + mutex_lock(_mutex);
>> if (ret == 0) {
>> - bitmap_set(cma->bitmap, pageno, count);
>> page = pfn_to_page(pfn);
>> break;
>> - } else if (ret != -EBUSY) {
>> +}
>> +bitmap_clear(cma->bitmap, pageno, count);
>> +if (ret != -EBUSY) {
>> break;
>> }
>> ...
>> mutex_unlock(_mutex);
>> pr_debug("%s(): returned %p\n", __func__, page);
>> return page;
>> }
Like Marek said, this will fail if two concurrent calls to
alloc_contig_range are made such that they operate on the same pageblock
(which is possible as the allocated regions do not need to be pageblock
aligned).
Another mutex could be added just for this one call, but as I understand
this would not solve the problem.
>> bool dma_release_from_contiguous(struct device *dev, struct page *pages,
>> int count)
>> {
>> ...
>> + free_contig_range(pfn, count);
>> mutex_lock(_mutex);
>> bitmap_clear(cma->bitmap, pfn - cma->base_pfn, count);
>> - free_contig_range(pfn, count);
>> mutex_unlock(_mutex);
>> ...
>> }
This *should* be fine. Didn't test it.
--
Best regards, _ _
.o. | Liege of Serenely Enlightened Majesty of o' \,=./ `o
..o | Computer Science, Micha? ?mina86? Nazarewicz(o o)
ooo +--ooO--(_)--Ooo--
-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL:
[BUG] Circular locking dependency - DRM/CMA/MM/hotplug/...
Hello,
On 2014-02-12 17:33, Russell King - ARM Linux wrote:
> On Wed, Feb 12, 2014 at 04:40:50PM +0100, Marek Szyprowski wrote:
> > Hello,
> >
> > On 2014-02-11 19:35, Russell King - ARM Linux wrote:
> >> The cubox-i4 just hit a new lockdep problem - not quite sure what to
> >> make of this - it looks like an interaction between quite a lot of
> >> locks - I suspect more than the lockdep code is reporting in its
> >> "Possible unsafe locking scenario" report.
> >>
> >> I'm hoping I've sent this to appropriate people... if anyone thinks
> >> this needs to go to someone else, please forward it. Thanks.
> >
> > From the attached log it looks like an issue (AB-BA deadlock) between
> > device mutex (>struct_mutex) and mm semaphore (>mmap_sem).
> > Similar issue has been discussed quite a long time ago in v4l2
> > subsystem:
>
> I think there's more locks involved than just those two.
>
> > https://www.mail-archive.com/linux-media at vger.kernel.org/msg38599.html
> > http://www.spinics.net/lists/linux-media/msg40225.html
> >
> > Solving it probably requires some changes in DRM core. I see no direct
> > relation between this issue and CMA itself.
>
> I don't think so - the locking in DRM is pretty sane. Let's take a
> look:
>
> >> the existing dependency chain (in reverse order) is:
> >> -> #5 (>struct_mutex){+.+...}:
> >> [] __lock_acquire+0x151c/0x1ca0
> >> [] lock_acquire+0xa0/0x130
> >> [] mutex_lock_nested+0x5c/0x3ac
> >> [] drm_gem_mmap+0x40/0xdc
> >> [] drm_gem_cma_mmap+0x14/0x2c
> >> [] mmap_region+0x3ac/0x59c
> >> [] do_mmap_pgoff+0x2c8/0x370
> >> [] vm_mmap_pgoff+0x6c/0x9c
> >> [] SyS_mmap_pgoff+0x54/0x98
> >> [] ret_fast_syscall+0x0/0x48
>
> vm_mmap_pgoff() takes mm->mmap_sem before calling do_mmap_pgoff().
> So, this results in the following locking order:
>
> mm->mmap_sem
> dev->struct_mutex
>
> >> -> #4 (>mmap_sem){++}:
> >> [] __lock_acquire+0x151c/0x1ca0
> >> [] lock_acquire+0xa0/0x130
> >> [] might_fault+0x6c/0x94
> >> [] con_set_unimap+0x158/0x27c
> >> [] vt_ioctl+0x1298/0x1388
> >> [] tty_ioctl+0x168/0xbf4
> >> [] do_vfs_ioctl+0x84/0x664
> >> [] SyS_ioctl+0x44/0x64
> >> [] ret_fast_syscall+0x0/0x48
>
> vt_ioctl() takes the console lock, so this results in:
>
> console_lock
> mm->mmap_sem
>
> >> -> #3 (console_lock){+.+.+.}:
> >> [] __lock_acquire+0x151c/0x1ca0
> >> [] lock_acquire+0xa0/0x130
> >> [] console_lock+0x60/0x74
> >> [] console_cpu_notify+0x28/0x34
> >> [] notifier_call_chain+0x4c/0x8c
> >> [] __raw_notifier_call_chain+0x1c/0x24
> >> [] __cpu_notify+0x34/0x50
> >> [] cpu_notify_nofail+0x18/0x24
> >> [] _cpu_down+0x100/0x244
> >> [] cpu_down+0x30/0x44
> >> [] cpu_subsys_offline+0x14/0x18
> >> [] device_offline+0x94/0xbc
> >> [] online_store+0x4c/0x74
> >> [] dev_attr_store+0x20/0x2c
> >> [] sysfs_kf_write+0x54/0x58
> >> [] kernfs_fop_write+0xc4/0x160
> >> [] vfs_write+0xbc/0x184
> >> [] SyS_write+0x48/0x70
> >> [] ret_fast_syscall+0x0/0x48
>
> cpu_down() takes cpu_hotplug.lock, so here we have:
>
> cpu_hotplug.lock
> console_lock
>
> >> -> #2 (cpu_hotplug.lock){+.+.+.}:
> >> [] __lock_acquire+0x151c/0x1ca0
> >> [] lock_acquire+0xa0/0x130
> >> [] mutex_lock_nested+0x5c/0x3ac
> >> [] get_online_cpus+0x3c/0x58
> >> [] lru_add_drain_all+0x24/0x190
> >> [] migrate_prep+0x10/0x18
> >> [] alloc_contig_range+0xf4/0x30c
> >> [] dma_alloc_from_contiguous+0x7c/0x130
> >> [] __alloc_from_contiguous+0x38/0x12c
> >> [] atomic_pool_init+0x74/0x128
> >> [] do_one_initcall+0x3c/0x164
> >> [] kernel_init_freeable+0x104/0x1d0
> >> [] kernel_init+0x10/0xec
> >> [] ret_from_fork+0x14/0x2c
>
> dma_alloc_from_contiguous takes the cma_mutex, so here we end up with:
>
> cma_mutex
> cpu_hotplug.lock
>
> >> -> #1 (lock){+.+...}:
> >> [] __lock_acquire+0x151c/0x1ca0
> >> [] lock_acquire+0xa0/0x130
> >> [] mutex_lock_nested+0x5c/0x3ac
> >> [] lru_add_drain_all+0x1c/0x190
> >> [] migrate_prep+0x10/0x18
> >> [] alloc_contig_range+0xf4/0x30c
> >> [] dma_alloc_from_contiguous+0x7c/0x130
> >> [] __alloc_from_contiguous+0x38/0x12c
> >> [] atomic_pool_init+0x74/0x128
> >> [] do_one_initcall+0x3c/0x164
> >> [] kernel_init_freeable+0x104/0x1d0
> >> [] kernel_init+0x10/0xec
> >> [] ret_from_fork+0x14/0x2c
>
> Ditto - here we have:
>
> cma_mutex
> lock
>
> where "lock" is nicely named... this is a lock inside lru_add_drain_all()
> and under this lock, we call get_online_cpus() and put_online_cpus().
> get_online_cpus() takes cpu_hotplug.lock, so
[BUG] Circular locking dependency - DRM/CMA/MM/hotplug/...
Please learn how to trim emails down to contain only the bits relevant to your reply, thanks. On Tue, Feb 18, 2014 at 03:25:21PM +0100, Marek Szyprowski wrote: > Hello, > > On 2014-02-12 17:33, Russell King - ARM Linux wrote: >> So, the full locking dependency tree is this: >> >> CPU0 CPU1CPU2CPU3CPU4 >> dev->struct_mutex (from #0) >> mm->mmap_sem >> dev->struct_mutex (from #5) >> console_lock (from #4) >> mm->mmap_sem >> cpu_hotplug.lock (from #3) >> console_lock >> cma_mutex (from >> #2, but also from #1) >> cpu_hotplug.lock >> cma_mutex >> >> Which is pretty sick - and I don't think that blaming this solely on V4L2 >> nor DRM is particularly fair. I believe the onus is on every author of >> one of those locks involved in that chain needs to re-analyse whether >> their locking is sane. >> >> For instance, what is cma_mutex protecting? Is it protecting the CMA >> bitmap? > > This lock is protecting CMA bitmap and also serializes all CMA allocations. > It is required by memory management core to serialize all calls to > alloc_contig_range() (otherwise page block's migrate types might get > overwritten). I don't see any other obvious solution for serializing > alloc_contig_range() calls. That's unfortunate, because what you're effectively asking is for every subsystem in the kernel to avoid a complex set of lock dependencies. It appears that two subsystems have now hit this, and I wouldn't be surprised if they weren't the last. > This will not work correctly if there will be 2 concurrent calls to > alloc_contig_range(), which will touch the same memory page blocks. Can you see any other way to lessen the impact of cma_mutex on the whole kernel? -- FTTC broadband for 0.8mile line: 5.8Mbps down 500kbps up. Estimation in database were 13.1 to 19Mbit for a good line, about 7.5+ for a bad. Estimate before purchase was "up to 13.2Mbit".
[BUG] Circular locking dependency - DRM/CMA/MM/hotplug/...
On Wed, Feb 12, 2014 at 04:33:17PM +, Russell King - ARM Linux wrote:
> On Wed, Feb 12, 2014 at 04:40:50PM +0100, Marek Szyprowski wrote:
> >> -> #3 (console_lock){+.+.+.}:
> >> [] __lock_acquire+0x151c/0x1ca0
> >> [] lock_acquire+0xa0/0x130
> >> [] console_lock+0x60/0x74
> >> [] console_cpu_notify+0x28/0x34
> >> [] notifier_call_chain+0x4c/0x8c
> >> [] __raw_notifier_call_chain+0x1c/0x24
> >> [] __cpu_notify+0x34/0x50
> >> [] cpu_notify_nofail+0x18/0x24
> >> [] _cpu_down+0x100/0x244
> >> [] cpu_down+0x30/0x44
> >> [] cpu_subsys_offline+0x14/0x18
> >> [] device_offline+0x94/0xbc
> >> [] online_store+0x4c/0x74
> >> [] dev_attr_store+0x20/0x2c
> >> [] sysfs_kf_write+0x54/0x58
> >> [] kernfs_fop_write+0xc4/0x160
> >> [] vfs_write+0xbc/0x184
> >> [] SyS_write+0x48/0x70
> >> [] ret_fast_syscall+0x0/0x48
>
> cpu_down() takes cpu_hotplug.lock, so here we have:
>
> cpu_hotplug.lock
> console_lock
The patche I've linked in my other mail will break the chain here, so
should solve this. And apparently with cpu hotplug we can hit this, too.
And having banged my head against the console_lock wall I think doing a
trylock here is generally the sanest option.
So imo we can just blame console_lock, not need to either beat up v4l,
drm, cma or anyone else really ;-)
Cheers, Daniel
--
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch
[BUG] Circular locking dependency - DRM/CMA/MM/hotplug/...
On Wed, Feb 12, 2014 at 07:29:01PM +0100, Daniel Vetter wrote:
> On Wed, Feb 12, 2014 at 04:33:17PM +, Russell King - ARM Linux wrote:
> > On Wed, Feb 12, 2014 at 04:40:50PM +0100, Marek Szyprowski wrote:
> > >> -> #3 (console_lock){+.+.+.}:
> > >> [] __lock_acquire+0x151c/0x1ca0
> > >> [] lock_acquire+0xa0/0x130
> > >> [] console_lock+0x60/0x74
> > >> [] console_cpu_notify+0x28/0x34
> > >> [] notifier_call_chain+0x4c/0x8c
> > >> [] __raw_notifier_call_chain+0x1c/0x24
> > >> [] __cpu_notify+0x34/0x50
> > >> [] cpu_notify_nofail+0x18/0x24
> > >> [] _cpu_down+0x100/0x244
> > >> [] cpu_down+0x30/0x44
> > >> [] cpu_subsys_offline+0x14/0x18
> > >> [] device_offline+0x94/0xbc
> > >> [] online_store+0x4c/0x74
> > >> [] dev_attr_store+0x20/0x2c
> > >> [] sysfs_kf_write+0x54/0x58
> > >> [] kernfs_fop_write+0xc4/0x160
> > >> [] vfs_write+0xbc/0x184
> > >> [] SyS_write+0x48/0x70
> > >> [] ret_fast_syscall+0x0/0x48
> >
> > cpu_down() takes cpu_hotplug.lock, so here we have:
> >
> > cpu_hotplug.lock
> > console_lock
>
> The patche I've linked in my other mail will break the chain here, so
> should solve this. And apparently with cpu hotplug we can hit this, too.
> And having banged my head against the console_lock wall I think doing a
> trylock here is generally the sanest option.
>
> So imo we can just blame console_lock, not need to either beat up v4l,
> drm, cma or anyone else really ;-)
I don't think CMA needs to hold its lock across the allocations/frees
though - given the size of this, I think it's best if /everyone/ tries
to reduce the locking interactions where possible.
The CMA issue needs to be done anyway - what it currently means is that
all CMA allocations in the kernel are serialised, even if an allocation
attempt sleeps, another allocation gets blocked.
So, sorting that out breaks the dependency there, and if it can be broken
elsewhere, that's an added bonus and will help prevent other issues
like this.
--
FTTC broadband for 0.8mile line: 5.8Mbps down 500kbps up. Estimation
in database were 13.1 to 19Mbit for a good line, about 7.5+ for a bad.
Estimate before purchase was "up to 13.2Mbit".
[BUG] Circular locking dependency - DRM/CMA/MM/hotplug/...
Hello,
On 2014-02-11 19:35, Russell King - ARM Linux wrote:
> The cubox-i4 just hit a new lockdep problem - not quite sure what to
> make of this - it looks like an interaction between quite a lot of
> locks - I suspect more than the lockdep code is reporting in its
> "Possible unsafe locking scenario" report.
>
> I'm hoping I've sent this to appropriate people... if anyone thinks
> this needs to go to someone else, please forward it. Thanks.
From the attached log it looks like an issue (AB-BA deadlock) between
device mutex (>struct_mutex) and mm semaphore (>mmap_sem).
Similar issue has been discussed quite a long time ago in v4l2
subsystem:
https://www.mail-archive.com/linux-media at vger.kernel.org/msg38599.html
http://www.spinics.net/lists/linux-media/msg40225.html
Solving it probably requires some changes in DRM core. I see no direct
relation between this issue and CMA itself.
> ==
> [ INFO: possible circular locking dependency detected ]
> 3.14.0-rc2+ #517 Tainted: GW
> ---
> Xorg/805 is trying to acquire lock:
> (cma_mutex){+.+.+.}, at: [] dma_release_from_contiguous+0xb8/0xf8
>
> but task is already holding lock:
> (>struct_mutex){+.+...}, at: []
> drm_gem_object_handle_unreference_unlocked+0xdc/0x148
>
> which lock already depends on the new lock.
>
>
> the existing dependency chain (in reverse order) is:
> -> #5 (>struct_mutex){+.+...}:
> [] __lock_acquire+0x151c/0x1ca0
> [] lock_acquire+0xa0/0x130
> [] mutex_lock_nested+0x5c/0x3ac
> [] drm_gem_mmap+0x40/0xdc
> [] drm_gem_cma_mmap+0x14/0x2c
> [] mmap_region+0x3ac/0x59c
> [] do_mmap_pgoff+0x2c8/0x370
> [] vm_mmap_pgoff+0x6c/0x9c
> [] SyS_mmap_pgoff+0x54/0x98
> [] ret_fast_syscall+0x0/0x48
> -> #4 (>mmap_sem){++}:
> [] __lock_acquire+0x151c/0x1ca0
> [] lock_acquire+0xa0/0x130
> [] might_fault+0x6c/0x94
> [] con_set_unimap+0x158/0x27c
> [] vt_ioctl+0x1298/0x1388
> [] tty_ioctl+0x168/0xbf4
> [] do_vfs_ioctl+0x84/0x664
> [] SyS_ioctl+0x44/0x64
> [] ret_fast_syscall+0x0/0x48
> -> #3 (console_lock){+.+.+.}:
> [] __lock_acquire+0x151c/0x1ca0
> [] lock_acquire+0xa0/0x130
> [] console_lock+0x60/0x74
> [] console_cpu_notify+0x28/0x34
> [] notifier_call_chain+0x4c/0x8c
> [] __raw_notifier_call_chain+0x1c/0x24
> [] __cpu_notify+0x34/0x50
> [] cpu_notify_nofail+0x18/0x24
> [] _cpu_down+0x100/0x244
> [] cpu_down+0x30/0x44
> [] cpu_subsys_offline+0x14/0x18
> [] device_offline+0x94/0xbc
> [] online_store+0x4c/0x74
> [] dev_attr_store+0x20/0x2c
> [] sysfs_kf_write+0x54/0x58
> [] kernfs_fop_write+0xc4/0x160
> [] vfs_write+0xbc/0x184
> [] SyS_write+0x48/0x70
> [] ret_fast_syscall+0x0/0x48
> -> #2 (cpu_hotplug.lock){+.+.+.}:
> [] __lock_acquire+0x151c/0x1ca0
> [] lock_acquire+0xa0/0x130
> [] mutex_lock_nested+0x5c/0x3ac
> [] get_online_cpus+0x3c/0x58
> [] lru_add_drain_all+0x24/0x190
> [] migrate_prep+0x10/0x18
> [] alloc_contig_range+0xf4/0x30c
> [] dma_alloc_from_contiguous+0x7c/0x130
> [] __alloc_from_contiguous+0x38/0x12c
> [] atomic_pool_init+0x74/0x128
> [] do_one_initcall+0x3c/0x164
> [] kernel_init_freeable+0x104/0x1d0
> [] kernel_init+0x10/0xec
> [] ret_from_fork+0x14/0x2c
> -> #1 (lock){+.+...}:
> [] __lock_acquire+0x151c/0x1ca0
> [] lock_acquire+0xa0/0x130
> [] mutex_lock_nested+0x5c/0x3ac
> [] lru_add_drain_all+0x1c/0x190
> [] migrate_prep+0x10/0x18
> [] alloc_contig_range+0xf4/0x30c
> [] dma_alloc_from_contiguous+0x7c/0x130
> [] __alloc_from_contiguous+0x38/0x12c
> [] atomic_pool_init+0x74/0x128
> [] do_one_initcall+0x3c/0x164
> [] kernel_init_freeable+0x104/0x1d0
> [] kernel_init+0x10/0xec
> [] ret_from_fork+0x14/0x2c
> -> #0 (cma_mutex){+.+.+.}:
> [] print_circular_bug+0x70/0x2f0
> [] __lock_acquire+0x1580/0x1ca0
> [] lock_acquire+0xa0/0x130
> [] mutex_lock_nested+0x5c/0x3ac
> [] dma_release_from_contiguous+0xb8/0xf8
> [] __arm_dma_free.isra.11+0x194/0x218
> [] arm_dma_free+0x1c/0x24
> [] drm_gem_cma_free_object+0x68/0xb8
> [] drm_gem_object_free+0x30/0x38
> [] drm_gem_object_handle_unreference_unlocked+0x108/0x148
> [] drm_gem_handle_delete+0xb0/0x10c
> [] drm_gem_dumb_destroy+0x14/0x18
> [] drm_mode_destroy_dumb_ioctl+0x34/0x40
> [] drm_ioctl+0x3f4/0x498
> [] do_vfs_ioctl+0x84/0x664
> [] SyS_ioctl+0x44/0x64
> [] ret_fast_syscall+0x0/0x48
>
> other info that might help us debug
[BUG] Circular locking dependency - DRM/CMA/MM/hotplug/...
On Wed, Feb 12, 2014 at 04:40:50PM +0100, Marek Szyprowski wrote:
> Hello,
>
> On 2014-02-11 19:35, Russell King - ARM Linux wrote:
>> The cubox-i4 just hit a new lockdep problem - not quite sure what to
>> make of this - it looks like an interaction between quite a lot of
>> locks - I suspect more than the lockdep code is reporting in its
>> "Possible unsafe locking scenario" report.
>>
>> I'm hoping I've sent this to appropriate people... if anyone thinks
>> this needs to go to someone else, please forward it. Thanks.
>
> From the attached log it looks like an issue (AB-BA deadlock) between
> device mutex (>struct_mutex) and mm semaphore (>mmap_sem).
> Similar issue has been discussed quite a long time ago in v4l2
> subsystem:
I think there's more locks involved than just those two.
> https://www.mail-archive.com/linux-media at vger.kernel.org/msg38599.html
> http://www.spinics.net/lists/linux-media/msg40225.html
>
> Solving it probably requires some changes in DRM core. I see no direct
> relation between this issue and CMA itself.
I don't think so - the locking in DRM is pretty sane. Let's take a
look:
>> the existing dependency chain (in reverse order) is:
>> -> #5 (>struct_mutex){+.+...}:
>> [] __lock_acquire+0x151c/0x1ca0
>> [] lock_acquire+0xa0/0x130
>> [] mutex_lock_nested+0x5c/0x3ac
>> [] drm_gem_mmap+0x40/0xdc
>> [] drm_gem_cma_mmap+0x14/0x2c
>> [] mmap_region+0x3ac/0x59c
>> [] do_mmap_pgoff+0x2c8/0x370
>> [] vm_mmap_pgoff+0x6c/0x9c
>> [] SyS_mmap_pgoff+0x54/0x98
>> [] ret_fast_syscall+0x0/0x48
vm_mmap_pgoff() takes mm->mmap_sem before calling do_mmap_pgoff().
So, this results in the following locking order:
mm->mmap_sem
dev->struct_mutex
>> -> #4 (>mmap_sem){++}:
>> [] __lock_acquire+0x151c/0x1ca0
>> [] lock_acquire+0xa0/0x130
>> [] might_fault+0x6c/0x94
>> [] con_set_unimap+0x158/0x27c
>> [] vt_ioctl+0x1298/0x1388
>> [] tty_ioctl+0x168/0xbf4
>> [] do_vfs_ioctl+0x84/0x664
>> [] SyS_ioctl+0x44/0x64
>> [] ret_fast_syscall+0x0/0x48
vt_ioctl() takes the console lock, so this results in:
console_lock
mm->mmap_sem
>> -> #3 (console_lock){+.+.+.}:
>> [] __lock_acquire+0x151c/0x1ca0
>> [] lock_acquire+0xa0/0x130
>> [] console_lock+0x60/0x74
>> [] console_cpu_notify+0x28/0x34
>> [] notifier_call_chain+0x4c/0x8c
>> [] __raw_notifier_call_chain+0x1c/0x24
>> [] __cpu_notify+0x34/0x50
>> [] cpu_notify_nofail+0x18/0x24
>> [] _cpu_down+0x100/0x244
>> [] cpu_down+0x30/0x44
>> [] cpu_subsys_offline+0x14/0x18
>> [] device_offline+0x94/0xbc
>> [] online_store+0x4c/0x74
>> [] dev_attr_store+0x20/0x2c
>> [] sysfs_kf_write+0x54/0x58
>> [] kernfs_fop_write+0xc4/0x160
>> [] vfs_write+0xbc/0x184
>> [] SyS_write+0x48/0x70
>> [] ret_fast_syscall+0x0/0x48
cpu_down() takes cpu_hotplug.lock, so here we have:
cpu_hotplug.lock
console_lock
>> -> #2 (cpu_hotplug.lock){+.+.+.}:
>> [] __lock_acquire+0x151c/0x1ca0
>> [] lock_acquire+0xa0/0x130
>> [] mutex_lock_nested+0x5c/0x3ac
>> [] get_online_cpus+0x3c/0x58
>> [] lru_add_drain_all+0x24/0x190
>> [] migrate_prep+0x10/0x18
>> [] alloc_contig_range+0xf4/0x30c
>> [] dma_alloc_from_contiguous+0x7c/0x130
>> [] __alloc_from_contiguous+0x38/0x12c
>> [] atomic_pool_init+0x74/0x128
>> [] do_one_initcall+0x3c/0x164
>> [] kernel_init_freeable+0x104/0x1d0
>> [] kernel_init+0x10/0xec
>> [] ret_from_fork+0x14/0x2c
dma_alloc_from_contiguous takes the cma_mutex, so here we end up with:
cma_mutex
cpu_hotplug.lock
>> -> #1 (lock){+.+...}:
>> [] __lock_acquire+0x151c/0x1ca0
>> [] lock_acquire+0xa0/0x130
>> [] mutex_lock_nested+0x5c/0x3ac
>> [] lru_add_drain_all+0x1c/0x190
>> [] migrate_prep+0x10/0x18
>> [] alloc_contig_range+0xf4/0x30c
>> [] dma_alloc_from_contiguous+0x7c/0x130
>> [] __alloc_from_contiguous+0x38/0x12c
>> [] atomic_pool_init+0x74/0x128
>> [] do_one_initcall+0x3c/0x164
>> [] kernel_init_freeable+0x104/0x1d0
>> [] kernel_init+0x10/0xec
>> [] ret_from_fork+0x14/0x2c
Ditto - here we have:
cma_mutex
lock
where "lock" is nicely named... this is a lock inside lru_add_drain_all()
and under this lock, we call get_online_cpus() and put_online_cpus().
get_online_cpus() takes cpu_hotplug.lock, so here we also have:
cma_mutex
lock
cpu_hotplug.lock
>> -> #0 (cma_mutex){+.+.+.}:
>> [] print_circular_bug+0x70/0x2f0
>> [] __lock_acquire+0x1580/0x1ca0
>> [] lock_acquire+0xa0/0x130
>> [] mutex_lock_nested+0x5c/0x3ac
>> []
[BUG] Circular locking dependency - DRM/CMA/MM/hotplug/...
On Tue, Feb 11, 2014 at 06:35:43PM +, Russell King - ARM Linux wrote:
> The cubox-i4 just hit a new lockdep problem - not quite sure what to
> make of this - it looks like an interaction between quite a lot of
> locks - I suspect more than the lockdep code is reporting in its
> "Possible unsafe locking scenario" report.
>
> I'm hoping I've sent this to appropriate people... if anyone thinks
> this needs to go to someone else, please forward it. Thanks.
I didn't dare to look too closely since the console_lock is in the chain,
but looks rather similar to
http://www.spinics.net/lists/kernel/msg1685909.html
akpm is still bikeshedding that one though.
-Daniel
>
> ==
> [ INFO: possible circular locking dependency detected ]
> 3.14.0-rc2+ #517 Tainted: GW
> ---
> Xorg/805 is trying to acquire lock:
> (cma_mutex){+.+.+.}, at: [] dma_release_from_contiguous+0xb8/0xf8
>
> but task is already holding lock:
> (>struct_mutex){+.+...}, at: []
> drm_gem_object_handle_unreference_unlocked+0xdc/0x148
>
> which lock already depends on the new lock.
>
>
> the existing dependency chain (in reverse order) is:
> -> #5 (>struct_mutex){+.+...}:
>[] __lock_acquire+0x151c/0x1ca0
>[] lock_acquire+0xa0/0x130
>[] mutex_lock_nested+0x5c/0x3ac
>[] drm_gem_mmap+0x40/0xdc
>[] drm_gem_cma_mmap+0x14/0x2c
>[] mmap_region+0x3ac/0x59c
>[] do_mmap_pgoff+0x2c8/0x370
>[] vm_mmap_pgoff+0x6c/0x9c
>[] SyS_mmap_pgoff+0x54/0x98
>[] ret_fast_syscall+0x0/0x48
> -> #4 (>mmap_sem){++}:
>[] __lock_acquire+0x151c/0x1ca0
>[] lock_acquire+0xa0/0x130
>[] might_fault+0x6c/0x94
>[] con_set_unimap+0x158/0x27c
>[] vt_ioctl+0x1298/0x1388
>[] tty_ioctl+0x168/0xbf4
>[] do_vfs_ioctl+0x84/0x664
>[] SyS_ioctl+0x44/0x64
>[] ret_fast_syscall+0x0/0x48
> -> #3 (console_lock){+.+.+.}:
>[] __lock_acquire+0x151c/0x1ca0
>[] lock_acquire+0xa0/0x130
>[] console_lock+0x60/0x74
>[] console_cpu_notify+0x28/0x34
>[] notifier_call_chain+0x4c/0x8c
>[] __raw_notifier_call_chain+0x1c/0x24
>[] __cpu_notify+0x34/0x50
>[] cpu_notify_nofail+0x18/0x24
>[] _cpu_down+0x100/0x244
>[] cpu_down+0x30/0x44
>[] cpu_subsys_offline+0x14/0x18
>[] device_offline+0x94/0xbc
>[] online_store+0x4c/0x74
>[] dev_attr_store+0x20/0x2c
>[] sysfs_kf_write+0x54/0x58
>[] kernfs_fop_write+0xc4/0x160
>[] vfs_write+0xbc/0x184
>[] SyS_write+0x48/0x70
>[] ret_fast_syscall+0x0/0x48
> -> #2 (cpu_hotplug.lock){+.+.+.}:
>[] __lock_acquire+0x151c/0x1ca0
>[] lock_acquire+0xa0/0x130
>[] mutex_lock_nested+0x5c/0x3ac
>[] get_online_cpus+0x3c/0x58
>[] lru_add_drain_all+0x24/0x190
>[] migrate_prep+0x10/0x18
>[] alloc_contig_range+0xf4/0x30c
>[] dma_alloc_from_contiguous+0x7c/0x130
>[] __alloc_from_contiguous+0x38/0x12c
>[] atomic_pool_init+0x74/0x128
>[] do_one_initcall+0x3c/0x164
>[] kernel_init_freeable+0x104/0x1d0
>[] kernel_init+0x10/0xec
>[] ret_from_fork+0x14/0x2c
> -> #1 (lock){+.+...}:
>[] __lock_acquire+0x151c/0x1ca0
>[] lock_acquire+0xa0/0x130
>[] mutex_lock_nested+0x5c/0x3ac
>[] lru_add_drain_all+0x1c/0x190
>[] migrate_prep+0x10/0x18
>[] alloc_contig_range+0xf4/0x30c
>[] dma_alloc_from_contiguous+0x7c/0x130
>[] __alloc_from_contiguous+0x38/0x12c
>[] atomic_pool_init+0x74/0x128
>[] do_one_initcall+0x3c/0x164
>[] kernel_init_freeable+0x104/0x1d0
>[] kernel_init+0x10/0xec
>[] ret_from_fork+0x14/0x2c
> -> #0 (cma_mutex){+.+.+.}:
>[] print_circular_bug+0x70/0x2f0
>[] __lock_acquire+0x1580/0x1ca0
>[] lock_acquire+0xa0/0x130
>[] mutex_lock_nested+0x5c/0x3ac
>[] dma_release_from_contiguous+0xb8/0xf8
>[] __arm_dma_free.isra.11+0x194/0x218
>[] arm_dma_free+0x1c/0x24
>[] drm_gem_cma_free_object+0x68/0xb8
>[] drm_gem_object_free+0x30/0x38
>[] drm_gem_object_handle_unreference_unlocked+0x108/0x148
>[] drm_gem_handle_delete+0xb0/0x10c
>[] drm_gem_dumb_destroy+0x14/0x18
>[] drm_mode_destroy_dumb_ioctl+0x34/0x40
>[] drm_ioctl+0x3f4/0x498
>[] do_vfs_ioctl+0x84/0x664
>[] SyS_ioctl+0x44/0x64
>[] ret_fast_syscall+0x0/0x48
>
> other info that might help us debug this:
>
> Chain exists of: cma_mutex --> >mmap_sem --> >struct_mutex
> Possible unsafe locking scenario:
>
>CPU0CPU1
>
> lock(>struct_mutex);
>lock(>mmap_sem);
>
[BUG] Circular locking dependency - DRM/CMA/MM/hotplug/...
The cubox-i4 just hit a new lockdep problem - not quite sure what to
make of this - it looks like an interaction between quite a lot of
locks - I suspect more than the lockdep code is reporting in its
"Possible unsafe locking scenario" report.
I'm hoping I've sent this to appropriate people... if anyone thinks
this needs to go to someone else, please forward it. Thanks.
==
[ INFO: possible circular locking dependency detected ]
3.14.0-rc2+ #517 Tainted: GW
---
Xorg/805 is trying to acquire lock:
(cma_mutex){+.+.+.}, at: [] dma_release_from_contiguous+0xb8/0xf8
but task is already holding lock:
(>struct_mutex){+.+...}, at: []
drm_gem_object_handle_unreference_unlocked+0xdc/0x148
which lock already depends on the new lock.
the existing dependency chain (in reverse order) is:
-> #5 (>struct_mutex){+.+...}:
[] __lock_acquire+0x151c/0x1ca0
[] lock_acquire+0xa0/0x130
[] mutex_lock_nested+0x5c/0x3ac
[] drm_gem_mmap+0x40/0xdc
[] drm_gem_cma_mmap+0x14/0x2c
[] mmap_region+0x3ac/0x59c
[] do_mmap_pgoff+0x2c8/0x370
[] vm_mmap_pgoff+0x6c/0x9c
[] SyS_mmap_pgoff+0x54/0x98
[] ret_fast_syscall+0x0/0x48
-> #4 (>mmap_sem){++}:
[] __lock_acquire+0x151c/0x1ca0
[] lock_acquire+0xa0/0x130
[] might_fault+0x6c/0x94
[] con_set_unimap+0x158/0x27c
[] vt_ioctl+0x1298/0x1388
[] tty_ioctl+0x168/0xbf4
[] do_vfs_ioctl+0x84/0x664
[] SyS_ioctl+0x44/0x64
[] ret_fast_syscall+0x0/0x48
-> #3 (console_lock){+.+.+.}:
[] __lock_acquire+0x151c/0x1ca0
[] lock_acquire+0xa0/0x130
[] console_lock+0x60/0x74
[] console_cpu_notify+0x28/0x34
[] notifier_call_chain+0x4c/0x8c
[] __raw_notifier_call_chain+0x1c/0x24
[] __cpu_notify+0x34/0x50
[] cpu_notify_nofail+0x18/0x24
[] _cpu_down+0x100/0x244
[] cpu_down+0x30/0x44
[] cpu_subsys_offline+0x14/0x18
[] device_offline+0x94/0xbc
[] online_store+0x4c/0x74
[] dev_attr_store+0x20/0x2c
[] sysfs_kf_write+0x54/0x58
[] kernfs_fop_write+0xc4/0x160
[] vfs_write+0xbc/0x184
[] SyS_write+0x48/0x70
[] ret_fast_syscall+0x0/0x48
-> #2 (cpu_hotplug.lock){+.+.+.}:
[] __lock_acquire+0x151c/0x1ca0
[] lock_acquire+0xa0/0x130
[] mutex_lock_nested+0x5c/0x3ac
[] get_online_cpus+0x3c/0x58
[] lru_add_drain_all+0x24/0x190
[] migrate_prep+0x10/0x18
[] alloc_contig_range+0xf4/0x30c
[] dma_alloc_from_contiguous+0x7c/0x130
[] __alloc_from_contiguous+0x38/0x12c
[] atomic_pool_init+0x74/0x128
[] do_one_initcall+0x3c/0x164
[] kernel_init_freeable+0x104/0x1d0
[] kernel_init+0x10/0xec
[] ret_from_fork+0x14/0x2c
-> #1 (lock){+.+...}:
[] __lock_acquire+0x151c/0x1ca0
[] lock_acquire+0xa0/0x130
[] mutex_lock_nested+0x5c/0x3ac
[] lru_add_drain_all+0x1c/0x190
[] migrate_prep+0x10/0x18
[] alloc_contig_range+0xf4/0x30c
[] dma_alloc_from_contiguous+0x7c/0x130
[] __alloc_from_contiguous+0x38/0x12c
[] atomic_pool_init+0x74/0x128
[] do_one_initcall+0x3c/0x164
[] kernel_init_freeable+0x104/0x1d0
[] kernel_init+0x10/0xec
[] ret_from_fork+0x14/0x2c
-> #0 (cma_mutex){+.+.+.}:
[] print_circular_bug+0x70/0x2f0
[] __lock_acquire+0x1580/0x1ca0
[] lock_acquire+0xa0/0x130
[] mutex_lock_nested+0x5c/0x3ac
[] dma_release_from_contiguous+0xb8/0xf8
[] __arm_dma_free.isra.11+0x194/0x218
[] arm_dma_free+0x1c/0x24
[] drm_gem_cma_free_object+0x68/0xb8
[] drm_gem_object_free+0x30/0x38
[] drm_gem_object_handle_unreference_unlocked+0x108/0x148
[] drm_gem_handle_delete+0xb0/0x10c
[] drm_gem_dumb_destroy+0x14/0x18
[] drm_mode_destroy_dumb_ioctl+0x34/0x40
[] drm_ioctl+0x3f4/0x498
[] do_vfs_ioctl+0x84/0x664
[] SyS_ioctl+0x44/0x64
[] ret_fast_syscall+0x0/0x48
other info that might help us debug this:
Chain exists of: cma_mutex --> >mmap_sem --> >struct_mutex
Possible unsafe locking scenario:
CPU0CPU1
lock(>struct_mutex);
lock(>mmap_sem);
lock(>struct_mutex);
lock(cma_mutex);
*** DEADLOCK ***
1 lock held by Xorg/805:
#0: (>struct_mutex){+.+...}, at: []
drm_gem_object_handle_unreference_unlocked+0xdc/0x148
stack backtrace:
CPU: 0 PID: 805 Comm: Xorg Tainted: GW3.14.0-rc2+ #517
Backtrace:
[] (dump_backtrace) from [] (show_stack+0x18/0x1c)
r6:c0a869f0 r5:c0a8d540 r4: r3:
[] (show_stack) from [] (dump_stack+0x70/0x8c)
[] (dump_stack) from [] (print_circular_bug+0x29c/0x2f0)
r4:c0a79570 r3:e9338980
[] (print_circular_bug) from []
