On Fri, Oct 3, 2014 at 1:13 AM, Gilles Chanteperdrix
<[email protected]> wrote:
> On 10/02/2014 05:52 PM, GP Orcullo wrote:
>> On Thu, Oct 2, 2014 at 9:36 PM, Gilles Chanteperdrix
>> <[email protected]> wrote:
>>> On 10/02/2014 03:27 PM, GP Orcullo wrote:
>>>> On Wed, Oct 1, 2014 at 5:20 PM, Gilles Chanteperdrix
>>>> <[email protected]> wrote:
>>>>> On 10/01/2014 11:12 AM, GP Orcullo wrote:
>>>>>> On Oct 1, 2014 3:54 PM, "Gilles Chanteperdrix" <
>>>>>> [email protected]> wrote:
>>>>>>>
>>>>>>> On 10/01/2014 01:32 AM, GP Orcullo wrote:
>>>>>>>> On Sep 30, 2014 8:16 PM, "Gilles Chanteperdrix" <
>>>>>>>> [email protected]> wrote:
>>>>>>>>>
>>>>>>>>> On 09/30/2014 02:04 PM, GP Orcullo wrote:
>>>>>>>>>> On Sep 30, 2014 7:30 PM, "Gilles Chanteperdrix" <
>>>>>>>>>> [email protected]> wrote:
>>>>>>>>>>>
>>>>>>>>>>> On 09/30/2014 07:31 AM, GP Orcullo wrote:
>>>>>>>>>>>> Hi,
>>>>>>>>>>>>
>>>>>>>>>>>> Running the switchtest for extended periods (>10 mins) causes the
>>>>>>>>>>>> machine to lockup.
>>>>>>>>>>>>
>>>>>>>>>>>> I'm running a modified xeno-regression-test which contains only the
>>>>>>>>>>>> following tests:
>>>>>>>>>>>>
>>>>>>>>>>>> check_alive /usr/lib/xenomai/testsuite/switchtest
>>>>>>>>>>>> check_alive /usr/lib/xenomai/testsuite/switchtest -s 1000
>>>>>>>>>>>> check_alive /usr/lib/xenomai/testsuite/latency ${1+"$@"}
>>>>>>>>>>>>
>>>>>>>>>>>> The script is invoked with the following arguments:
>>>>>>>>>>>>
>>>>>>>>>>>> nohup sudo ./xeno-regression-test -l
>>>>>>>>>>>> "/usr/lib/xenomai/testsuite/dohell -m /media/work 36000" -t 2 >
>>>>>>>>>>>> /dev/null & top -d0.5
>>>>>>>>>>>>
>>>>>>>>>>>> The kernel dumps the OOPS information intermittently so it's
>>>>>> difficult
>>>>>>>>>>>> to diagnose the issue.
>>>>>>>>>>>>
>>>>>>>>>>>> Attached is the kernel config and the logfile.
>>>>>>>>>>>
>>>>>>>>>>> Ok, this is an exynos. Sorry, but I have never seen the patch for
>>>>>>>>>>> exynos, so I do not know what is inside. You should direct your
>>>>>>>>>>> questions to whoever provided you with this support.
>>>>>>>>>>
>>>>>>>>>> I'm in the process of porting xenomai to run on exynos.
>>>>>>>>>>
>>>>>>>>>> The ipipe-core-3.8.13-arm-3.patch applies cleanly to the 3.8.13.11
>>>>>>>> kernel
>>>>>>>>>> used by the odroid U3 board.
>>>>>>>>>>
>>>>>>>>>> Attached is the ipipe patch that I've made.
>>>>>>>>>>
>>>>>>>>>> I was just wondering what would cause switchtest to fail. The error
>>>>>>>> that I
>>>>>>>>>> can see is that the system is running out of memory and I don't know
>>>>>>>>>> exactly what is causing this.
>>>>>>>>>
>>>>>>>>> Certainly not switchtest as it does not do any memory allocation.
>>>>>>>>> However, the dohell script has a loop creating a large file and
>>>>>> removing
>>>>>>>>> it. So, could you try and run the dohell script with an unpatched
>>>>>> kernel
>>>>>>>>> and see if you have the error?
>>>>>>>>>
>>>>>>>>
>>>>>>>> Running dohell on a patched and unpatched kernel doesn't trigger the
>>>>>> lockup.
>>>>>>>>
>>>>>>>> Running switchtest without dohell works OK.
>>>>>>>
>>>>>>> Is the problem a lockup, or an OOM?
>>>>>>>
>>>>>>
>>>>>> It's a lockup.
>>>>>>
>>>>>> The OOM message is the only one that I've captured so far. Most of the
>>>>>> time the kernel doesn't spew any messages before the lockup.
>>>>>>
>>>>>> The lockups are repeatable but generating any error messages isn't.
>>>>>
>>>>> Are you running the tests on the serial console, or with ssh? Do you
>>>>> have unlocked context switch enabled? Have you tried enabling some debug
>>>>> options?
>>>>>
>>>>
>>>> I'm using the serial console to log the kernel messages and ssh to run
>>>> the command. Using purely the serial console has the same results.
>>>
>>> The main point was to avoid redirecting standard error to /dev/null to
>>> see any application error message. Doing this on the serial console may
>>> be a better idea that on ssh, because it means you are less likely to
>>> miss a message that would be sent just prior to the system dying.
>>>
>>>>
>>>> Is this the context switch?: "CONFIG_XENO_HW_UNLOCKED_SWITCH=y"
>>>
>>> Yes, please try to disable it if you have it enabled.
>>>
>>>>
>>>> I will try playing again with the debug options and see if I can get
>>>> something useful.
>>>>
>>>>> Also note that xeno-regression-test puts the system under a lot of
>>>>> stress, so it may happen that there is no output for some time (several
>>>>> minutes), normally the test should stop by itself if there is no output
>>>>> for something like 30 minutes. So, I would recommend not redirecting
>>>>> xeno-test output to see if there is any error before the lockup, and
>>>>> when you see the lockup, leave the system for 30 minutes to see if it
>>>>> does not restart or if xeno-regression-test can exit gracefully.
>>>>>
>>>>
>>>> This is a total lockup. There's a heartbeat led that dies when it occurs.
>>>
>>> Well the heartbeat led does not prove anything: some Linux kernel
>>> activity can very well prevent it from being toggled. Say if for
>>> instance it is toggled by a thread and the activity that hogs the kernel
>>> is a softirq that never ends.
>>>
>>>>
>>>> Attached is one error log that I had captured previously and this one
>>>> had the CONFIG_CPU_IDLE enabled. I've lost track on which kernel this
>>>> trace came from but maybe the error looks familiar.
>>>
>>> This trace misses an important information: the reason for the error.
>>> So, please capture the serial console to a file, and post the complete
>>> file, from boot up to the error.
>>>
>>> Anyway, you did not answered my question: did you try to leave the
>>> system on for say 30 minutes of 1 hour after the lockup to see if it
>>> does not recover?
>>>
>>>
>>
>> The system never recovered.
>>
>> With the context switch disabled, I was able to capture this error:
>>
>> [ 210.482299] INFO: rcu_preempt detected stalls on CPUs/tasks:)
>> [ 210.487790] Task dump for CPU 2:
>> [ 210.490995] switchtest R running 0 3915 3639 0x00000002
>> [ 210.497340] [<c0453ddc>] (__schedule+0x1fc/0x5f8) from [<00000010>] (0x10)
>> [ 390.507943] INFO: rcu_preempt detected stalls on CPUs/tasks: { 2}
>> (detected )
>> [ 390.513510] Task dump for CPU 2:
>> [ 390.516716] switchtest R running 0 3915 3639 0x00000002
>> [ 390.523065] [<c0453ddc>] (__schedule+0x1fc/0x5f8) from [<00000010>] (0x10)
>>
>> <c0453ddc> points to the following section:
>>
>> #ifndef __ARCH_WANT_UNLOCKED_CTXSW
>> spin_release(&rq->lock.dep_map, 1, _THIS_IP_);
>> c0453dc8: ebf04b13 bl c0066a1c <lock_release>
>> #endif
>>
>> context_tracking_task_switch(prev, next);
>
> You do not have context tracking enabled, right?
>
>
> --
> Gilles.
The system managed to dump some error messages.
--
GP Orcullo
-------------- next part --------------
[ 407.688809] INFO: rcu_preempt detected stalls on CPUs/tasks: { 3} (detected
by 0, t=12002 jiffies, g=22447, c=22446, q=113448)
[ 407.694560] Task dump for CPU 3:
[ 407.697763] switchtest R running 0 3979 3640 0x00000002
[ 407.704111] [<c0453ddc>] (__schedule+0x1fc/0x5f8) from [<00000010>] (0x10)
[ 587.714458] INFO: rcu_preempt detected stalls on CPUs/tasks: { 3} (detected
by 2, t=48007 jiffies, g=22447, c=22446, q=443504)
[ 587.720205] Task dump for CPU 3:
[ 587.723407] switchtest R running 0 3979 3640 0x00000002
[ 587.729755] [<c0453ddc>] (__schedule+0x1fc/0x5f8) from [<00000010>] (0x10)
[ 767.740128] INFO: rcu_preempt detected stalls on CPUs/tasks: { 3} (detected
by 2, t=84012 jiffies, g=22447, c=22446, q=1136198)
[ 767.745964] Task dump for CPU 3:
[ 767.749167] switchtest R running 0 3979 3640 0x00000002
[ 767.755515] [<c0453ddc>] (__schedule+0x1fc/0x5f8) from [<00000010>] (0x10)
[ 782.763027] ------------[ cut here ]------------
[ 782.763099] WARNING: at include/linux/ipipe_domain.h:212
__ipipe_dispatch_irq+0x35c/0x5a0()
[ 782.770380] Modules linked in: nfsv3 nfsv4 nfsd exportfs nfs_acl auth_rpcgss
nfs lockd sunrpc vfat fat smsc95xx usbnet mii gpio_keys
[ 782.782218] [<c0015624>] (unwind_backtrace+0x0/0xf8) from [<c0026118>]
(warn_slowpath_common+0x4c/0x64)
[ 782.791606] [<c0026118>] (warn_slowpath_common+0x4c/0x64) from [<c002614c>]
(warn_slowpath_null+0x1c/0x24)
[ 782.801232] [<c002614c>] (warn_slowpath_null+0x1c/0x24) from [<c00886ac>]
(__ipipe_dispatch_irq+0x35c/0x5a0)
[ 782.811035] [<c00886ac>] (__ipipe_dispatch_irq+0x35c/0x5a0) from
[<c00200c8>] (combiner_handle_cascade_irq+0x80/0xf4)
[ 782.821620] [<c00200c8>] (combiner_handle_cascade_irq+0x80/0xf4) from
[<c0019110>] (__ipipe_ack_irq+0xc/0x10)
[ 782.831510] [<c0019110>] (__ipipe_ack_irq+0xc/0x10) from [<c0088594>]
(__ipipe_dispatch_irq+0x244/0x5a0)
[ 782.840956] [<c0088594>] (__ipipe_dispatch_irq+0x244/0x5a0) from
[<c0008508>] (__ipipe_grab_irq+0x48/0xd0)
[ 782.850582] [<c0008508>] (__ipipe_grab_irq+0x48/0xd0) from [<c0008790>]
(gic_handle_irq+0x34/0x68)
[ 782.859512] Exception stack(0xe6131c78 to 0xe6131cc0)
[ 782.864539] 1c60:
c001f9f8 c006cadc
[ 782.872699] 1c80: 20000053 ffffffff e6131cc4 c000e540 f8820000 00000004
01512000 00000001
[ 782.880848] 1ca0: c1b11180 c06021c0 00000000 00000003 c01322b8 00000001
00000000 c062af44
[ 782.889000] [<c0008790>] (gic_handle_irq+0x34/0x68) from [<c000e540>]
(__irq_svc+0x40/0x6c)
[ 782.897323] Exception stack(0xe6131c90 to 0xe6131cd8)
[ 782.902350] 1c80: f8820000 00000004
01512000 00000001
[ 782.910508] 1ca0: c1b11180 c06021c0 00000000 00000003 c01322b8 00000001
00000000 c062af44
[ 782.918659] 1cc0: 00000004 e6131cd8 c001f9f8 c006cadc 20000053 ffffffff
[ 782.925253] [<c000e540>] (__irq_svc+0x40/0x6c) from [<c006cadc>]
(smp_call_function_many+0x1ac/0x288)
[ 782.934456] [<c006cadc>] (smp_call_function_many+0x1ac/0x288) from
[<c006cbf0>] (on_each_cpu_mask+0x38/0x98)
[ 782.944251] [<c006cbf0>] (on_each_cpu_mask+0x38/0x98) from [<c0132aa0>]
(__alloc_pages_nodemask+0x5ec/0x954)
[ 782.954076] [<c0132aa0>] (__alloc_pages_nodemask+0x5ec/0x954) from
[<bf0983cc>] (nfs_readdir_xdr_to_array+0xb4/0x234 [nfs])
[ 782.965212] [<bf0983cc>] (nfs_readdir_xdr_to_array+0xb4/0x234 [nfs]) from
[<bf098568>] (nfs_readdir_filler+0x1c/0x84 [nfs])
[ 782.976270] [<bf098568>] (nfs_readdir_filler+0x1c/0x84 [nfs]) from
[<c012c274>] (do_read_cache_page+0x70/0x15c)
[ 782.986311] [<c012c274>] (do_read_cache_page+0x70/0x15c) from [<c012c3a4>]
(read_cache_page_async+0x18/0x20)
[ 782.996112] [<c012c3a4>] (read_cache_page_async+0x18/0x20) from [<c012c3b4>]
(read_cache_page+0x8/0x10)
[ 783.005494] [<c012c3b4>] (read_cache_page+0x8/0x10) from [<bf0986e0>]
(nfs_readdir+0x110/0x554 [nfs])
[ 783.014692] [<bf0986e0>] (nfs_readdir+0x110/0x554 [nfs]) from [<c0175b38>]
(vfs_readdir+0x80/0xa4)
[ 783.023617] [<c0175b38>] (vfs_readdir+0x80/0xa4) from [<c0175d04>]
(sys_getdents64+0x64/0xd8)
[ 783.032103] [<c0175d04>] (sys_getdents64+0x64/0xd8) from [<c000e9c0>]
(ret_fast_syscall+0x0/0x34)
[ 783.040951] ---[ end trace 0054ffe34e60e6b0 ]---
[ 783.045534] ------------[ cut here ]------------
[ 783.050118] WARNING: at include/linux/ipipe_domain.h:153
__ipipe_set_irq_pending+0xe8/0x16c()
[ 783.058637] Modules linked in: nfsv3 nfsv4 nfsd exportfs nfs_acl auth_rpcgss
nfs lockd sunrpc vfat fat smsc95xx usbnet mii gpio_keys
[ 783.070497] [<c0015624>] (unwind_backtrace+0x0/0xf8) from [<c0026118>]
(warn_slowpath_common+0x4c/0x64)
[ 783.079893] [<c0026118>] (warn_slowpath_common+0x4c/0x64) from [<c002614c>]
(warn_slowpath_null+0x1c/0x24)
[ 783.089529] [<c002614c>] (warn_slowpath_null+0x1c/0x24) from [<c00861ac>]
(__ipipe_set_irq_pending+0xe8/0x16c)
[ 783.099495] [<c00861ac>] (__ipipe_set_irq_pending+0xe8/0x16c) from
[<c0088544>] (__ipipe_dispatch_irq+0x1f4/0x5a0)
[ 783.109815] [<c0088544>] (__ipipe_dispatch_irq+0x1f4/0x5a0) from
[<c00200c8>] (combiner_handle_cascade_irq+0x80/0xf4)
[ 783.120407] [<c00200c8>] (combiner_handle_cascade_irq+0x80/0xf4) from
[<c0019110>] (__ipipe_ack_irq+0xc/0x10)
[ 783.130283] [<c0019110>] (__ipipe_ack_irq+0xc/0x10) from [<c0088594>]
(__ipipe_dispatch_irq+0x244/0x5a0)
[ 783.139733] [<c0088594>] (__ipipe_dispatch_irq+0x244/0x5a0) from
[<c0008508>] (__ipipe_grab_irq+0x48/0xd0)
[ 783.149363] [<c0008508>] (__ipipe_grab_irq+0x48/0xd0) from [<c0008790>]
(gic_handle_irq+0x34/0x68)
[ 783.158286] Exception stack(0xe6131c78 to 0xe6131cc0)
[ 783.163312] 1c60:
c001f9f8 c006cadc
[ 783.171470] 1c80: 20000053 ffffffff e6131cc4 c000e540 f8820000 00000004
01512000 00000001
[ 783.179625] 1ca0: c1b11180 c06021c0 00000000 00000003 c01322b8 00000001
00000000 c062af44
[ 783.187777] [<c0008790>] (gic_handle_irq+0x34/0x68) from [<c000e540>]
(__irq_svc+0x40/0x6c)
[ 783.196098] Exception stack(0xe6131c90 to 0xe6131cd8)
[ 783.201125] 1c80: f8820000 00000004
01512000 00000001
[ 783.209283] 1ca0: c1b11180 c06021c0 00000000 00000003 c01322b8 00000001
00000000 c062af44
[ 783.217435] 1cc0: 00000004 e6131cd8 c001f9f8 c006cadc 20000053 ffffffff
[ 783.224027] [<c000e540>] (__irq_svc+0x40/0x6c) from [<c006cadc>]
(smp_call_function_many+0x1ac/0x288)
[ 783.233225] [<c006cadc>] (smp_call_function_many+0x1ac/0x288) from
[<c006cbf0>] (on_each_cpu_mask+0x38/0x98)
[ 783.243026] [<c006cbf0>] (on_each_cpu_mask+0x38/0x98) from [<c0132aa0>]
(__alloc_pages_nodemask+0x5ec/0x954)
[ 783.252851] [<c0132aa0>] (__alloc_pages_nodemask+0x5ec/0x954) from
[<bf0983cc>] (nfs_readdir_xdr_to_array+0xb4/0x234 [nfs])
[ 783.263958] [<bf0983cc>] (nfs_readdir_xdr_to_array+0xb4/0x234 [nfs]) from
[<bf098568>] (nfs_readdir_filler+0x1c/0x84 [nfs])
[ 783.275048] [<bf098568>] (nfs_readdir_filler+0x1c/0x84 [nfs]) from
[<c012c274>] (do_read_cache_page+0x70/0x15c)
[ 783.285090] [<c012c274>] (do_read_cache_page+0x70/0x15c) from [<c012c3a4>]
(read_cache_page_async+0x18/0x20)
[ 783.294889] [<c012c3a4>] (read_cache_page_async+0x18/0x20) from [<c012c3b4>]
(read_cache_page+0x8/0x10)
[ 783.304269] [<c012c3b4>] (read_cache_page+0x8/0x10) from [<bf0986e0>]
(nfs_readdir+0x110/0x554 [nfs])
[ 783.313463] [<bf0986e0>] (nfs_readdir+0x110/0x554 [nfs]) from [<c0175b38>]
(vfs_readdir+0x80/0xa4)
[ 783.322391] [<c0175b38>] (vfs_readdir+0x80/0xa4) from [<c0175d04>]
(sys_getdents64+0x64/0xd8)
[ 783.330878] [<c0175d04>] (sys_getdents64+0x64/0xd8) from [<c000e9c0>]
(ret_fast_syscall+0x0/0x34)
[ 783.339722] ---[ end trace 0054ffe34e60e6b1 ]---
[ 783.344309] ------------[ cut here ]------------
[ 783.348911] WARNING: at kernel/ipipe/core.c:582
__ipipe_set_irq_pending+0x150/0x16c()
[ 783.356721] Modules linked in: nfsv3 nfsv4 nfsd exportfs nfs_acl auth_rpcgss
nfs lockd sunrpc vfat fat smsc95xx usbnet mii gpio_keys
[ 783.368577] [<c0015624>] (unwind_backtrace+0x0/0xf8) from [<c0026118>]
(warn_slowpath_common+0x4c/0x64)
[ 783.377972] [<c0026118>] (warn_slowpath_common+0x4c/0x64) from [<c002614c>]
(warn_slowpath_null+0x1c/0x24)
[ 783.387599] [<c002614c>] (warn_slowpath_null+0x1c/0x24) from [<c0086214>]
(__ipipe_set_irq_pending+0x150/0x16c)
[ 783.397663] [<c0086214>] (__ipipe_set_irq_pending+0x150/0x16c) from
[<c0088544>] (__ipipe_dispatch_irq+0x1f4/0x5a0)
[ 783.408067] [<c0088544>] (__ipipe_dispatch_irq+0x1f4/0x5a0) from
[<c00200c8>] (combiner_handle_cascade_irq+0x80/0xf4)
[ 783.418649] [<c00200c8>] (combiner_handle_cascade_irq+0x80/0xf4) from
[<c0019110>] (__ipipe_ack_irq+0xc/0x10)
[ 783.428535] [<c0019110>] (__ipipe_ack_irq+0xc/0x10) from [<c0088594>]
(__ipipe_dispatch_irq+0x244/0x5a0)
[ 783.437985] [<c0088594>] (__ipipe_dispatch_irq+0x244/0x5a0) from
[<c0008508>] (__ipipe_grab_irq+0x48/0xd0)
[ 783.447613] [<c0008508>] (__ipipe_grab_irq+0x48/0xd0) from [<c0008790>]
(gic_handle_irq+0x34/0x68)
[ 783.456541] Exception stack(0xe6131c78 to 0xe6131cc0)
[ 783.461573] 1c60:
c001f9f8 c006cadc
[ 783.469729] 1c80: 20000053 ffffffff e6131cc4 c000e540 f8820000 00000004
01512000 00000001
[ 783.477878] 1ca0: c1b11180 c06021c0 00000000 00000003 c01322b8 00000001
00000000 c062af44
[ 783.486031] [<c0008790>] (gic_handle_irq+0x34/0x68) from [<c000e540>]
(__irq_svc+0x40/0x6c)
[ 783.494353] Exception stack(0xe6131c90 to 0xe6131cd8)
[ 783.499395] 1c80: f8820000 00000004
01512000 00000001
[ 783.507538] 1ca0: c1b11180 c06021c0 00000000 00000003 c01322b8 00000001
00000000 c062af44
[ 783.515690] 1cc0: 00000004 e6131cd8 c001f9f8 c006cadc 20000053 ffffffff
[ 783.522283] [<c000e540>] (__irq_svc+0x40/0x6c) from [<c006cadc>]
(smp_call_function_many+0x1ac/0x288)
[ 783.531479] [<c006cadc>] (smp_call_function_many+0x1ac/0x288) from
[<c006cbf0>] (on_each_cpu_mask+0x38/0x98)
[ 783.541280] [<c006cbf0>] (on_each_cpu_mask+0x38/0x98) from [<c0132aa0>]
(__alloc_pages_nodemask+0x5ec/0x954)
[ 783.551099] [<c0132aa0>] (__alloc_pages_nodemask+0x5ec/0x954) from
[<bf0983cc>] (nfs_readdir_xdr_to_array+0xb4/0x234 [nfs])
[ 783.562218] [<bf0983cc>] (nfs_readdir_xdr_to_array+0xb4/0x234 [nfs]) from
[<bf098568>] (nfs_readdir_filler+0x1c/0x84 [nfs])
[ 783.573302] [<bf098568>] (nfs_readdir_filler+0x1c/0x84 [nfs]) from
[<c012c274>] (do_read_cache_page+0x70/0x15c)
[ 783.583342] [<c012c274>] (do_read_cache_page+0x70/0x15c) from [<c012c3a4>]
(read_cache_page_async+0x18/0x20)
[ 783.593142] [<c012c3a4>] (read_cache_page_async+0x18/0x20) from [<c012c3b4>]
(read_cache_page+0x8/0x10)
[ 783.602523] [<c012c3b4>] (read_cache_page+0x8/0x10) from [<bf0986e0>]
(nfs_readdir+0x110/0x554 [nfs])
[ 783.611719] [<bf0986e0>] (nfs_readdir+0x110/0x554 [nfs]) from [<c0175b38>]
(vfs_readdir+0x80/0xa4)
[ 783.620634] [<c0175b38>] (vfs_readdir+0x80/0xa4) from [<c0175d04>]
(sys_getdents64+0x64/0xd8)
[ 783.629133] [<c0175d04>] (sys_getdents64+0x64/0xd8) from [<c000e9c0>]
(ret_fast_syscall+0x0/0x34)
[ 783.637974] ---[ end trace 0054ffe34e60e6b2 ]---
[ 800.110222] BUG: soft lockup - CPU#0 stuck for 23s! [ls:3654]
[ 800.110322] Modules linked in: nfsv3 nfsv4 nfsd exportfs nfs_acl auth_rpcgss
nfs lockd sunrpc vfat fat smsc95xx usbnet mii gpio_keys
[ 800.122197]
[ 800.123674] Pid: 3654, comm: ls
[ 800.128362] CPU: 0 Tainted: G W (3.8.13.11-xen-deb #21)
[ 800.134784] PC is at s3c24xx_serial_console_putchar+0x38/0x74
[ 800.140505] LR is at 0x0
[ 800.143016] pc : [<c0290da4>] lr : [<00000000>] psr: 60000153
[ 800.143016] sp : 00000000 ip : 00000000 fp : 00000000
[ 800.154472] r10: 00000000 r9 : 00000000 r8 : 00000000
[ 800.159669] r7 : 00000000 r6 : 00000000 r5 : 00000000 r4 : 00000000
[ 800.166174] r3 : 00000000 r2 : 00000000 r1 : 00000000 r0 : 00000000
[ 800.170226] BUG: soft lockup - CPU#2 stuck for 23s! [ps:17584]
[ 800.170252] Modules linked in: nfsv3 nfsv4 nfsd exportfs nfs_acl auth_rpcgss
nfs lockd sunrpc vfat fat smsc95xx usbnet mii gpio_keys
[ 800.170254]
[ 800.170256] Pid: 17584, comm: ps
[ 800.170259] CPU: 2 Tainted: G W (3.8.13.11-xen-deb #21)
[ 800.170265] PC is at smp_call_function_many+0x1a8/0x288
[ 800.170268] LR is at 0x0
[ 800.170272] pc : [<c006cad8>] lr : [<00000000>] psr: 20000053
[ 800.170272] sp : 00000000 ip : 00000000 fp : 00000000
[ 800.170274] r10: 00000000 r9 : 00000000 r8 : 00000000
[ 800.170277] r7 : 00000000 r6 : 00000000 r5 : 00000000 r4 : 00000000
[ 800.170279] r3 : 00000000 r2 : 00000000 r1 : 00000000 r0 : 00000000
[ 800.170283] Flags: nzCv IRQs on FIQs off Mode SVC_32 ISA ARM Segment
user
[ 800.170286] Control: 10c5387d Table: 65fe404a DAC: 00000015
[ 800.170298] [<c0015624>] (unwind_backtrace+0x0/0xf8) from [<c007c138>]
(watchdog_timer_fn+0x134/0x17c)
[ 800.170309] [<c007c138>] (watchdog_timer_fn+0x134/0x17c) from [<c0046ba8>]
(__run_hrtimer.isra.19+0x44/0xd0)
[ 800.170316] [<c0046ba8>] (__run_hrtimer.isra.19+0x44/0xd0) from [<c0047318>]
(hrtimer_interrupt+0x108/0x2b4)
[ 800.170324] [<c0047318>] (hrtimer_interrupt+0x108/0x2b4) from [<c0020f30>]
(exynos4_mct_tick_isr+0x28/0x34)
[ 800.170331] [<c0020f30>] (exynos4_mct_tick_isr+0x28/0x34) from [<c007f6dc>]
(handle_percpu_devid_irq+0x44/0xa0)
[ 800.170338] [<c007f6dc>] (handle_percpu_devid_irq+0x44/0xa0) from
[<c007c3bc>] (generic_handle_irq+0x18/0x2c)
[ 800.170346] [<c007c3bc>] (generic_handle_irq+0x18/0x2c) from [<c000f918>]
(handle_IRQ+0x38/0x94)
[ 800.170354] [<c000f918>] (handle_IRQ+0x38/0x94) from [<c0087b54>]
(__ipipe_do_sync_stage+0x1a8/0x2c0)
[ 800.170360] [<c0087b54>] (__ipipe_do_sync_stage+0x1a8/0x2c0) from
[<c0008508>] (__ipipe_grab_irq+0x48/0xd0)
[ 800.170365] [<c0008508>] (__ipipe_grab_irq+0x48/0xd0) from [<c0008790>]
(gic_handle_irq+0x34/0x68)
[ 800.170368] Exception stack(0xe5fe9dd0 to 0xe5fe9e18)
[ 800.170372] 9dc0: c001f9f8 c006cad8
20000053 ffffffff
[ 800.170378] 9de0: e5fe9e1c c000e540 f8828000 00000004 01526000 00000001
c1b25180 c06021c0
[ 800.170383] 9e00: 00000000 00000003 c01322b8 00000001 00000000 c062af44
[ 800.170389] [<c0008790>] (gic_handle_irq+0x34/0x68) from [<c000e540>]
(__irq_svc+0x40/0x6c)
[ 800.170391] Exception stack(0xe5fe9de8 to 0xe5fe9e30)
[ 800.170396] 9de0: f8828000 00000004 01526000 00000001
c1b25180 c06021c0
[ 800.170402] 9e00: 00000000 00000003 c01322b8 00000001 00000000 c062af44
00000004 e5fe9e30
[ 800.170406] 9e20: c001f9f8 c006cad8 20000053 ffffffff
[ 800.170413] [<c000e540>] (__irq_svc+0x40/0x6c) from [<c006cad8>]
(smp_call_function_many+0x1a8/0x288)
[ 800.170419] [<c006cad8>] (smp_call_function_many+0x1a8/0x288) from
[<c006cbf0>] (on_each_cpu_mask+0x38/0x98)
[ 800.170426] [<c006cbf0>] (on_each_cpu_mask+0x38/0x98) from [<c0132aa0>]
(__alloc_pages_nodemask+0x5ec/0x954)
[ 800.170432] [<c0132aa0>] (__alloc_pages_nodemask+0x5ec/0x954) from
[<c0132e1c>] (__get_free_pages+0x14/0x2c)
[ 800.170440] [<c0132e1c>] (__get_free_pages+0x14/0x2c) from [<c01b0d5c>]
(proc_info_read+0x40/0xdc)
[ 800.170450] [<c01b0d5c>] (proc_info_read+0x40/0xdc) from [<c0165ecc>]
(vfs_read+0x9c/0x140)
[ 800.170457] [<c0165ecc>] (vfs_read+0x9c/0x140) from [<c0165fac>]
(sys_read+0x3c/0x70)
[ 800.170464] [<c0165fac>] (sys_read+0x3c/0x70) from [<c000e9c0>]
(ret_fast_syscall+0x0/0x34)
[ 800.483079] Flags: nZCv IRQs on FIQs off Mode SVC_32 ISA ARM Segment
user
[ 800.490277] Control: 10c5387d Table: 6614804a DAC: 00000015
[ 800.496006] [<c0015624>] (unwind_backtrace+0x0/0xf8) from [<c007c138>]
(watchdog_timer_fn+0x134/0x17c)
[ 800.505289] [<c007c138>] (watchdog_timer_fn+0x134/0x17c) from [<c0046ba8>]
(__run_hrtimer.isra.19+0x44/0xd0)
[ 800.515088] [<c0046ba8>] (__run_hrtimer.isra.19+0x44/0xd0) from [<c0047318>]
(hrtimer_interrupt+0x108/0x2b4)
[ 800.524889] [<c0047318>] (hrtimer_interrupt+0x108/0x2b4) from [<c0020f30>]
(exynos4_mct_tick_isr+0x28/0x34)
[ 800.534601] [<c0020f30>] (exynos4_mct_tick_isr+0x28/0x34) from [<c007f6dc>]
(handle_percpu_devid_irq+0x44/0xa0)
[ 800.544662] [<c007f6dc>] (handle_percpu_devid_irq+0x44/0xa0) from
[<c007c3bc>] (generic_handle_irq+0x18/0x2c)
[ 800.554549] [<c007c3bc>] (generic_handle_irq+0x18/0x2c) from [<c000f918>]
(handle_IRQ+0x38/0x94)
[ 800.563308] [<c000f918>] (handle_IRQ+0x38/0x94) from [<c0087b54>]
(__ipipe_do_sync_stage+0x1a8/0x2c0)
[ 800.572500] [<c0087b54>] (__ipipe_do_sync_stage+0x1a8/0x2c0) from
[<c0008508>] (__ipipe_grab_irq+0x48/0xd0)
[ 800.582213] [<c0008508>] (__ipipe_grab_irq+0x48/0xd0) from [<c0008790>]
(gic_handle_irq+0x34/0x68)
[ 800.591141] Exception stack(0xe6131c78 to 0xe6131cc0)
[ 800.596168] 1c60:
c001f9f8 c006cad8
[ 800.604327] 1c80: 20000053 ffffffff e6131cc4 c000e540 f8820000 00000004
01512000 00000001
[ 800.612479] 1ca0: c1b11180 c06021c0 00000000 00000003 c01322b8 00000001
00000000 c062af44
[ 800.620632] [<c0008790>] (gic_handle_irq+0x34/0x68) from [<c000e540>]
(__irq_svc+0x40/0x6c)
[ 800.628954] Exception stack(0xe6131c90 to 0xe6131cd8)
[ 800.633982] 1c80: f8820000 00000004
01512000 00000001
[ 800.642139] 1ca0: c1b11180 c06021c0 00000000 00000003 c01322b8 00000001
00000000 c062af44
[ 800.650291] 1cc0: 00000004 e6131cd8 c001f9f8 c006cad8 20000053 ffffffff
[ 800.656883] [<c000e540>] (__irq_svc+0x40/0x6c) from [<c006cad8>]
(smp_call_function_many+0x1a8/0x288)
[ 800.666080] [<c006cad8>] (smp_call_function_many+0x1a8/0x288) from
[<c006cbf0>] (on_each_cpu_mask+0x38/0x98)
[ 800.675880] [<c006cbf0>] (on_each_cpu_mask+0x38/0x98) from [<c0132aa0>]
(__alloc_pages_nodemask+0x5ec/0x954)
[ 800.685701] [<c0132aa0>] (__alloc_pages_nodemask+0x5ec/0x954) from
[<bf0983cc>] (nfs_readdir_xdr_to_array+0xb4/0x234 [nfs])
[ 800.696813] [<bf0983cc>] (nfs_readdir_xdr_to_array+0xb4/0x234 [nfs]) from
[<bf098568>] (nfs_readdir_filler+0x1c/0x84 [nfs])
[ 800.707899] [<bf098568>] (nfs_readdir_filler+0x1c/0x84 [nfs]) from
[<c012c274>] (do_read_cache_page+0x70/0x15c)
[ 800.717942] [<c012c274>] (do_read_cache_page+0x70/0x15c) from [<c012c3a4>]
(read_cache_page_async+0x18/0x20)
[ 800.727741] [<c012c3a4>] (read_cache_page_async+0x18/0x20) from [<c012c3b4>]
(read_cache_page+0x8/0x10)
[ 800.737123] [<c012c3b4>] (read_cache_page+0x8/0x10) from [<bf0986e0>]
(nfs_readdir+0x110/0x554 [nfs])
[ 800.746318] [<bf0986e0>] (nfs_readdir+0x110/0x554 [nfs]) from [<c0175b38>]
(vfs_readdir+0x80/0xa4)
[ 800.755234] [<c0175b38>] (vfs_readdir+0x80/0xa4) from [<c0175d04>]
(sys_getdents64+0x64/0xd8)
[ 800.763733] [<c0175d04>] (sys_getdents64+0x64/0xd8) from [<c000e9c0>]
(ret_fast_syscall+0x0/0x34)
[ 804.140245] BUG: soft lockup - CPU#1 stuck for 22s! [ls:5936]
[ 804.140347] Modules linked in: nfsv3 nfsv4 nfsd exportfs nfs_acl auth_rpcgss
nfs lockd sunrpc vfat fat smsc95xx usbnet mii gpio_keys
[ 804.152222]
[ 804.153699] Pid: 5936, comm: ls
[ 804.158386] CPU: 1 Tainted: G W (3.8.13.11-xen-deb #21)
[ 804.164807] PC is at smp_call_function_many+0x1a4/0x288
[ 804.170008] LR is at 0x0
[ 804.172521] pc : [<c006cad4>] lr : [<00000000>] psr: 20000053
[ 804.172521] sp : 00000000 ip : 00000000 fp : 00000000
[ 804.183977] r10: 00000000 r9 : 00000000 r8 : 00000000
[ 804.189174] r7 : 00000000 r6 : 00000000 r5 : 00000000 r4 : 00000000
[ 804.195680] r3 : 00000000 r2 : 00000000 r1 : 00000000 r0 : 00000000
[ 804.202185] Flags: nzCv IRQs on FIQs off Mode SVC_32 ISA ARM Segment
user
[ 804.209385] Control: 10c5387d Table: 65af004a DAC: 00000015
[ 804.215113] [<c0015624>] (unwind_backtrace+0x0/0xf8) from [<c007c138>]
(watchdog_timer_fn+0x134/0x17c)
[ 804.224396] [<c007c138>] (watchdog_timer_fn+0x134/0x17c) from [<c0046ba8>]
(__run_hrtimer.isra.19+0x44/0xd0)
[ 804.234195] [<c0046ba8>] (__run_hrtimer.isra.19+0x44/0xd0) from [<c0047318>]
(hrtimer_interrupt+0x108/0x2b4)
[ 804.243997] [<c0047318>] (hrtimer_interrupt+0x108/0x2b4) from [<c0020f30>]
(exynos4_mct_tick_isr+0x28/0x34)
[ 804.253708] [<c0020f30>] (exynos4_mct_tick_isr+0x28/0x34) from [<c007f6dc>]
(handle_percpu_devid_irq+0x44/0xa0)
[ 804.263769] [<c007f6dc>] (handle_percpu_devid_irq+0x44/0xa0) from
[<c007c3bc>] (generic_handle_irq+0x18/0x2c)
[ 804.273656] [<c007c3bc>] (generic_handle_irq+0x18/0x2c) from [<c000f918>]
(handle_IRQ+0x38/0x94)
[ 804.282415] [<c000f918>] (handle_IRQ+0x38/0x94) from [<c0087b54>]
(__ipipe_do_sync_stage+0x1a8/0x2c0)
[ 804.291607] [<c0087b54>] (__ipipe_do_sync_stage+0x1a8/0x2c0) from
[<c0008508>] (__ipipe_grab_irq+0x48/0xd0)
[ 804.301320] [<c0008508>] (__ipipe_grab_irq+0x48/0xd0) from [<c0008790>]
(gic_handle_irq+0x34/0x68)
[ 804.310249] Exception stack(0xe5b87c78 to 0xe5b87cc0)
[ 804.315276] 7c60:
c001f9f8 c006cad4
[ 804.323434] 7c80: 20000053 ffffffff e5b87cc4 c000e540 f8824000 00000004
0151c000 00000001
[ 804.331586] 7ca0: c1b1b180 c06021c0 00000000 00000002 c01322b8 00000001
00000000 c062af44
[ 804.339739] [<c0008790>] (gic_handle_irq+0x34/0x68) from [<c000e540>]
(__irq_svc+0x40/0x6c)
[ 804.348062] Exception stack(0xe5b87c90 to 0xe5b87cd8)
[ 804.353089] 7c80: f8824000 00000004
0151c000 00000001
[ 804.361247] 7ca0: c1b1b180 c06021c0 00000000 00000002 c01322b8 00000001
00000000 c062af44
[ 804.369398] 7cc0: 00000004 e5b87cd8 c001f9f8 c006cad4 20000053 ffffffff
[ 804.375990] [<c000e540>] (__irq_svc+0x40/0x6c) from [<c006cad4>]
(smp_call_function_many+0x1a4/0x288)
[ 804.385186] [<c006cad4>] (smp_call_function_many+0x1a4/0x288) from
[<c006cbf0>] (on_each_cpu_mask+0x38/0x98)
[ 804.394987] [<c006cbf0>] (on_each_cpu_mask+0x38/0x98) from [<c0132aa0>]
(__alloc_pages_nodemask+0x5ec/0x954)
[ 804.404807] [<c0132aa0>] (__alloc_pages_nodemask+0x5ec/0x954) from
[<bf0983cc>] (nfs_readdir_xdr_to_array+0xb4/0x234 [nfs])
[ 804.415918] [<bf0983cc>] (nfs_readdir_xdr_to_array+0xb4/0x234 [nfs]) from
[<bf098568>] (nfs_readdir_filler+0x1c/0x84 [nfs])
[ 804.427004] [<bf098568>] (nfs_readdir_filler+0x1c/0x84 [nfs]) from
[<c012c274>] (do_read_cache_page+0x70/0x15c)
[ 804.437049] [<c012c274>] (do_read_cache_page+0x70/0x15c) from [<c012c3a4>]
(read_cache_page_async+0x18/0x20)
[ 804.446848] [<c012c3a4>] (read_cache_page_async+0x18/0x20) from [<c012c3b4>]
(read_cache_page+0x8/0x10)
[ 804.456228] [<c012c3b4>] (read_cache_page+0x8/0x10) from [<bf0986e0>]
(nfs_readdir+0x110/0x554 [nfs])
[ 804.465423] [<bf0986e0>] (nfs_readdir+0x110/0x554 [nfs]) from [<c0175b38>]
(vfs_readdir+0x80/0xa4)
[ 804.474341] [<c0175b38>] (vfs_readdir+0x80/0xa4) from [<c0175d04>]
(sys_getdents64+0x64/0xd8)
[ 804.482841] [<c0175d04>] (sys_getdents64+0x64/0xd8) from [<c000e9c0>]
(ret_fast_syscall+0x0/0x34)
[ 947.765771] INFO: rcu_preempt detected stalls on CPUs/tasks: { 3} (detected
by 1, t=120017 jiffies, g=22447, c=22446, q=1166139)
[ 947.771694] Task dump for CPU 3:
[ 947.774897] switchtest R running 0 3979 3640 0x00000002
[ 947.781244] [<c0453ddc>] (__schedule+0x1fc/0x5f8) from [<00000010>] (0x10)
[ 1127.791434] INFO: rcu_preempt detected stalls on CPUs/tasks: { 3} (detected
by 0, t=156022 jiffies, g=22447, c=22446, q=1166139)
[ 1127.797358] Task dump for CPU 3:
[ 1127.800561] switchtest R running 0 3979 3640 0x00000002
[ 1127.806908] [<c0453ddc>] (__schedule+0x1fc/0x5f8) from [<00000010>] (0x10)
[ 1307.817087] INFO: rcu_preempt detected stalls on CPUs/tasks: { 3} (detected
by 0, t=192027 jiffies, g=22447, c=22446, q=1166139)
[ 1307.823002] Task dump for CPU 3:
[ 1307.826206] switchtest R running 0 3979 3640 0x00000002
[ 1307.832553] [<c0453ddc>] (__schedule+0x1fc/0x5f8) from [<00000010>] (0x10)
[ 1487.842745] INFO: rcu_preempt detected stalls on CPUs/tasks: { 3} (detected
by 0, t=228032 jiffies, g=22447, c=22446, q=1166139)
[ 1487.848665] Task dump for CPU 3:
[ 1487.851868] switchtest R running 0 3979 3640 0x00000002
[ 1487.858213] [<c0453ddc>] (__schedule+0x1fc/0x5f8) from [<00000010>] (0x10)
[ 1667.868406] INFO: rcu_preempt detected stalls on CPUs/tasks: { 3} (detected
by 0, t=264037 jiffies, g=22447, c=22446, q=1166139)
[ 1667.874328] Task dump for CPU 3:
[ 1667.877531] switchtest R running 0 3979 3640 0x00000002
[ 1667.883878] [<c0453ddc>] (__schedule+0x1fc/0x5f8) from [<00000010>] (0x10)
[ 1847.894064] INFO: rcu_preempt detected stalls on CPUs/tasks: { 3} (detected
by 2, t=300042 jiffies, g=22447, c=22446, q=1166139)
[ 1847.899981] Task dump for CPU 3:
[ 1847.903185] switchtest R running 0 3979 3640 0x00000002
[ 1847.909531] [<c0453ddc>] (__schedule+0x1fc/0x5f8) from [<00000010>] (0x10)
[ 2027.919723] INFO: rcu_preempt detected stalls on CPUs/tasks: { 3} (detected
by 0, t=336047 jiffies, g=22447, c=22446, q=1166139)
[ 2027.925645] Task dump for CPU 3:
[ 2027.928848] switchtest R running 0 3979 3640 0x00000002
[ 2027.935194] [<c0453ddc>] (__schedule+0x1fc/0x5f8) from [<00000010>] (0x10)
_______________________________________________
Xenomai mailing list
[email protected]
http://www.xenomai.org/mailman/listinfo/xenomai
