On Wed, 20 Jun 2001, Marcelo Tosatti wrote:
> > happened again (vt1 and 2 echo but shells are unresponsive, vt3+ don't
> > echo) only active process was the program allocating 192mb and writing to
> > it, no find this time.
>
> Can you get the backtrace of this process?
the offending process is memeat (allocates 192*1024*1024 bytes, then
memsets it to 0, then sleeps forever, though apparently it never gets to
that part). It has to be in memset(ptr, 0, 192*1024*1024).
ksymoops from sysrq-m attached, and vmstat from just before the hang and
just after.
justin
free sibling
task PC stack pid father child younger older
init R current 0 1 0 1312 (NOTLB)
Call Trace: [<c01279a1>] [<c0127b06>] [<c012863a>] [<c01286dc>] [<c012588b>]
[<c0125a1f>] [<c0136ca9>] [<c0137bbd>] [<c0134d4e>] [<c0106aeb>]
keventd S FFFFFFFF 0 2 1 3 (L-TLB)
Call Trace: [<c011d2eb>] [<c010545c>]
kswapd R C1451FA8 0 3 1 4 2 (L-TLB)
Call Trace: [<c0110cbb>] [<c0110c00>] [<c011122a>] [<c0127aa1>] [<c010545c>]
kreclaimd S 00000286 0 4 1 5 3 (L-TLB)
Call Trace: [<c01111d5>] [<c0127b52>] [<c010545c>]
bdflush S 00000282 0 5 1 6 4 (L-TLB)
Call Trace: [<c01111d5>] [<c0131144>] [<c010545c>]
kupdated S C1479FC8 0 6 1 8 5 (L-TLB)
Call Trace: [<c0110cbb>] [<c0110c00>] [<c01311b9>] [<c010545c>]
kreiserfsd S CFAB9FB4 0 8 1 12 6 (L-TLB)
Call Trace: [<c0110cbb>] [<c0110c00>] [<c011122a>] [<c016b6f7>] [<c010545c>]
devfsd S CF706000 8 12 1 22 8 (NOTLB)
Call Trace: [<c015126c>] [<c015d890>] [<c015e145>] [<c0164cf7>] [<c016551f>]
[<c0194725>] [<c0165a22>] [<c0190948>] [<c01909a2>] [<c0157395>] [<c015758d>]
[<c0164b20>] [<c0157a79>] [<c0164cf7>] [<c016551f>] [<c013457c>] [<c012e040>]
[<c0158b16>] [<c0158f12>] [<c0164cf7>] [<c016551f>] [<d081a044>] [<c016c87a>]
[<c016cc09>] [<c016c256>] [<c0159073>] [<c0136e95>] [<c0134e14>] [<c012d7b6>]
[<c0106aeb>]
klogd S 7FFFFFFF 0 22 1 27 12 (NOTLB)
Call Trace: [<c0110c5f>] [<c01da0af>] [<c01da967>] [<c01a8345>] [<c01a856e>]
[<c012d87b>] [<c0106aeb>]
syslogd R 00000000 0 27 1 30 22 (NOTLB)
Call Trace: [<c012858c>] [<c01286dc>] [<c013ae57>] [<c01acdf0>] [<c01a874f>]
[<c013b036>] [<c013b490>] [<c0106aeb>]
kapm-idled S CF23FF94 0 30 1 53 27 (L-TLB)
Call Trace: [<c0110cbb>] [<c0110c00>] [<d090a9d7>] [<d090bcdc>] [<d090c08c>]
[<d090c08c>] [<d090b2e5>] [<c0105453>] [<c010545c>]
cardmgr S 7FFFFFFF 4812 53 1 72 30 (NOTLB)
Call Trace: [<c0110c5f>] [<c019ff9c>] [<c013b0f1>] [<c013b490>] [<c0106aeb>]
sshd S 7FFFFFFF 0 72 1 74 53 (NOTLB)
Call Trace: [<c0110c5f>] [<c013b0f1>] [<c013b490>] [<c0106aeb>]
xfs S CE65FF2C 0 74 1 78 72 (NOTLB)
Call Trace: [<c0110cbb>] [<c0110c00>] [<c013b0f1>] [<c013b490>] [<c0106aeb>]
zsh S CF94FFB0 0 78 1 1338 79 74 (NOTLB)
Call Trace: [<c0105cc7>] [<fffafefe>] [<c0106aeb>]
agetty S 7FFFFFFF 1172 79 1 81 78 (NOTLB)
Call Trace: [<c0110c5f>] [<c017754d>] [<c0173738>] [<c012d7b6>] [<c0106aeb>]
zsh S 7FFFFFFF 0 81 1 122 79 (NOTLB)
Call Trace: [<c0110c5f>] [<c017754d>] [<c0173738>] [<c012d7b6>] [<c0106aeb>]
zsh S CF1BDFB0 0 122 1 138 148 81 (NOTLB)
Call Trace: [<c0105cc7>] [<fffafefe>] [<c0106aeb>]
zsh S C30FFFB0 4708 138 122 150 (NOTLB)
Call Trace: [<c0105cc7>] [<fffafefe>] [<c0106aeb>]
gpm S C32D3F2C 4640 148 1 151 122 (NOTLB)
Call Trace: [<c0110cbb>] [<c0110c00>] [<c013b0f1>] [<c013b490>] [<c0106aeb>]
[<c010002b>]
ssh S 7FFFFFFF 0 150 138 (NOTLB)
Call Trace: [<c0110c5f>] [<c013b0f1>] [<c013b490>] [<c0106aeb>]
agetty S 7FFFFFFF 0 151 1 1312 148 (NOTLB)
Call Trace: [<c0110c5f>] [<c017754d>] [<c0173738>] [<c012d7b6>] [<c0106aeb>]
zsh S C9BCDFB0 24 1312 1 1431 151 (NOTLB)
Call Trace: [<c0105cc7>] [<fffafefe>] [<c0106aeb>]
zsh S C1AB2000 0 1319 78 1432 1338 (NOTLB)
Call Trace: [<c013614c>] [<c0136223>] [<c012d7b6>] [<c0106aeb>]
tail S CE86FF8C 24 1338 78 1319 (NOTLB)
Call Trace: [<c0110cbb>] [<c0110c00>] [<c0119a48>] [<c0106aeb>]
memeat R 00000000 376 1431 1312 (NOTLB)
Call Trace: [<c012858c>] [<c011f0be>] [<c011f14b>] [<c011f25b>] [<c0110374>]
[<c01104d4>] [<c0110374>] [<c01aae1b>] [<c01194f9>] [<c01193d7>] [<c01195c4>]
[<c010a9f8>] [<c0110f3c>] [<c0110374>] [<c0106c18>]
zsh R 00000000 4600 1432 1319 (NOTLB)
Call Trace: [<c012858c>] [<c011eba2>] [<c011f291>] [<c0110374>] [<c01104d4>]
[<c0110374>] [<c0128c75>] [<c012d47c>] [<c0106c18>]
ksymoops 2.4.1 on i686 2.4.5-ac16. Options used
-V (default)
-k /proc/ksyms (default)
-l /proc/modules (default)
-o /lib/modules/2.4.5-ac16/ (default)
-m /boot/System.map (specified)
Call Trace: [<c01279a1>] [<c0127b06>] [<c012863a>] [<c01286dc>] [<c012588b>]
[<c0125a1f>] [<c0136ca9>] [<c0137bbd>] [<c0134d4e>] [<c0106aeb>]
Call Trace: [<c011d2eb>] [<c010545c>]
Call Trace: [<c0110cbb>] [<c0110c00>] [<c011122a>] [<c0127aa1>] [<c010545c>]
Call Trace: [<c01111d5>] [<c0127b52>] [<c010545c>]
Call Trace: [<c01111d5>] [<c0131144>] [<c010545c>]
Call Trace: [<c0110cbb>] [<c0110c00>] [<c01311b9>] [<c010545c>]
Call Trace: [<c0110cbb>] [<c0110c00>] [<c011122a>] [<c016b6f7>] [<c010545c>]
Call Trace: [<c015126c>] [<c015d890>] [<c015e145>] [<c0164cf7>] [<c016551f>]
[<c0194725>] [<c0165a22>] [<c0190948>] [<c01909a2>] [<c0157395>] [<c015758d>]
[<c0164b20>] [<c0157a79>] [<c0164cf7>] [<c016551f>] [<c013457c>] [<c012e040>]
[<c0158b16>] [<c0158f12>] [<c0164cf7>] [<c016551f>] [<d081a044>] [<c016c87a>]
[<c016cc09>] [<c016c256>] [<c0159073>] [<c0136e95>] [<c0134e14>] [<c012d7b6>]
[<c0106aeb>]
Call Trace: [<c0110c5f>] [<c01da0af>] [<c01da967>] [<c01a8345>] [<c01a856e>]
[<c012d87b>] [<c0106aeb>]
Call Trace: [<c012858c>] [<c01286dc>] [<c013ae57>] [<c01acdf0>] [<c01a874f>]
[<c013b036>] [<c013b490>] [<c0106aeb>]
Call Trace: [<c0110cbb>] [<c0110c00>] [<d090a9d7>] [<d090bcdc>] [<d090c08c>]
[<d090c08c>] [<d090b2e5>] [<c0105453>] [<c010545c>]
Call Trace: [<c0110c5f>] [<c019ff9c>] [<c013b0f1>] [<c013b490>] [<c0106aeb>]
Call Trace: [<c0110c5f>] [<c013b0f1>] [<c013b490>] [<c0106aeb>]
Call Trace: [<c0110cbb>] [<c0110c00>] [<c013b0f1>] [<c013b490>] [<c0106aeb>]
Call Trace: [<c0105cc7>] [<fffafefe>] [<c0106aeb>]
Call Trace: [<c0110c5f>] [<c017754d>] [<c0173738>] [<c012d7b6>] [<c0106aeb>]
Call Trace: [<c0110c5f>] [<c017754d>] [<c0173738>] [<c012d7b6>] [<c0106aeb>]
Call Trace: [<c0105cc7>] [<fffafefe>] [<c0106aeb>]
Call Trace: [<c0105cc7>] [<fffafefe>] [<c0106aeb>]
Call Trace: [<c0110cbb>] [<c0110c00>] [<c013b0f1>] [<c013b490>] [<c0106aeb>]
[<c010002b>]
Call Trace: [<c0110c5f>] [<c013b0f1>] [<c013b490>] [<c0106aeb>]
Call Trace: [<c0110c5f>] [<c017754d>] [<c0173738>] [<c012d7b6>] [<c0106aeb>]
Call Trace: [<c0105cc7>] [<fffafefe>] [<c0106aeb>]
Call Trace: [<c013614c>] [<c0136223>] [<c012d7b6>] [<c0106aeb>]
Call Trace: [<c0110cbb>] [<c0110c00>] [<c0119a48>] [<c0106aeb>]
Call Trace: [<c012858c>] [<c011f0be>] [<c011f14b>] [<c011f25b>] [<c0110374>]
[<c01104d4>] [<c0110374>] [<c01aae1b>] [<c01194f9>] [<c01193d7>] [<c01195c4>]
[<c010a9f8>] [<c0110f3c>] [<c0110374>] [<c0106c18>]
Call Trace: [<c012858c>] [<c011eba2>] [<c011f291>] [<c0110374>] [<c01104d4>]
[<c0110374>] [<c0128c75>] [<c012d47c>] [<c0106c18>]
Warning (Oops_read): Code line not seen, dumping what data is available
Trace; c01279a1 <do_try_to_free_pages+19/58>
Trace; c0127b06 <try_to_free_pages+22/2c>
Trace; c012863a <__alloc_pages+1b2/240>
Trace; c01286dc <__get_free_pages+14/20>
Trace; c012588b <kmem_cache_grow+af/1fc>
Trace; c0125a1f <kmem_cache_alloc+47/54>
Trace; c0136ca9 <getname+19/98>
Trace; c0137bbd <__user_walk+11/58>
Trace; c0134d4e <sys_stat64+16/78>
Trace; c0106aeb <system_call+33/38>
Trace; c011d2eb <context_thread+ef/194>
Trace; c010545c <kernel_thread+28/38>
Trace; c0110cbb <schedule_timeout+73/94>
Trace; c0110c00 <process_timeout+0/48>
Trace; c011122a <interruptible_sleep_on_timeout+42/5c>
Trace; c0127aa1 <kswapd+c1/e0>
Trace; c010545c <kernel_thread+28/38>
Trace; c01111d5 <interruptible_sleep_on+3d/50>
Trace; c0127b52 <kreclaimd+42/a0>
Trace; c010545c <kernel_thread+28/38>
Trace; c01111d5 <interruptible_sleep_on+3d/50>
Trace; c0131144 <bdflush+ac/b0>
Trace; c010545c <kernel_thread+28/38>
Trace; c0110cbb <schedule_timeout+73/94>
Trace; c0110c00 <process_timeout+0/48>
Trace; c01311b9 <kupdate+71/e8>
Trace; c010545c <kernel_thread+28/38>
Trace; c0110cbb <schedule_timeout+73/94>
Trace; c0110c00 <process_timeout+0/48>
Trace; c011122a <interruptible_sleep_on_timeout+42/5c>
Trace; c016b6f7 <reiserfs_journal_commit_thread+8f/d4>
Trace; c010545c <kernel_thread+28/38>
Trace; c015126c <devfsd_read+f4/3a8>
Trace; c015d890 <reiserfs_kfree+14/38>
Trace; c015e145 <unfix_nodes+155/160>
Trace; c0164cf7 <is_tree_node+37/54>
Trace; c016551f <search_by_key+80b/c64>
Trace; c0194725 <ide_dmaproc+13d/210>
Trace; c0165a22 <search_for_position_by_key+aa/380>
Trace; c0190948 <start_request+148/210>
Trace; c01909a2 <start_request+1a2/210>
Trace; c0157395 <make_cpu_key+39/40>
Trace; c015758d <_get_block_create_0+9d/5bc>
Trace; c0164b20 <pathrelse+1c/2c>
Trace; c0157a79 <_get_block_create_0+589/5bc>
Trace; c0164cf7 <is_tree_node+37/54>
Trace; c016551f <search_by_key+80b/c64>
Trace; c013457c <cdget+40/bc>
Trace; c012e040 <init_special_inode+44/d4>
Trace; c0158b16 <init_inode+1f2/1fc>
Trace; c0158f12 <reiserfs_read_inode2+be/c8>
Trace; c0164cf7 <is_tree_node+37/54>
Trace; c016551f <search_by_key+80b/c64>
Trace; d081a044 <_end+10569398/106593b4>
Trace; c016c87a <check_journal_end+20e/23c>
Trace; c016cc09 <do_journal_end+ad/a60>
Trace; c016c256 <journal_end+16/1c>
Trace; c0159073 <reiserfs_dirty_inode+4b/54>
Trace; c0136e95 <path_release+d/2c>
Trace; c0134e14 <sys_lstat64+64/70>
Trace; c012d7b6 <sys_read+96/cc>
Trace; c0106aeb <system_call+33/38>
Trace; c0110c5f <schedule_timeout+17/94>
Trace; c01da0af <unix_wait_for_peer+9b/c0>
Trace; c01da967 <unix_dgram_sendmsg+2c3/370>
Trace; c01a8345 <sock_sendmsg+69/88>
Trace; c01a856e <sock_write+b2/bc>
Trace; c012d87b <sys_write+8f/c4>
Trace; c0106aeb <system_call+33/38>
Trace; c012858c <__alloc_pages+104/240>
Trace; c01286dc <__get_free_pages+14/20>
Trace; c013ae57 <__pollwait+33/8c>
Trace; c01acdf0 <datagram_poll+24/e4>
Trace; c01a874f <sock_poll+23/28>
Trace; c013b036 <do_select+e2/1dc>
Trace; c013b490 <sys_select+338/474>
Trace; c0106aeb <system_call+33/38>
Trace; c0110cbb <schedule_timeout+73/94>
Trace; c0110c00 <process_timeout+0/48>
Trace; d090a9d7 <[apm]apm_mainloop+e3/100>
Trace; d090bcdc <[apm]error_table+4f8/85b>
Trace; d090c08c <[apm]apm_waitqueue+4/c>
Trace; d090c08c <[apm]apm_waitqueue+4/c>
Trace; d090b2e5 <[apm]apm+295/2a8>
Trace; c0105453 <kernel_thread+1f/38>
Trace; c010545c <kernel_thread+28/38>
Trace; c0110c5f <schedule_timeout+17/94>
Trace; c019ff9c <ds_poll+64/78>
Trace; c013b0f1 <do_select+19d/1dc>
Trace; c013b490 <sys_select+338/474>
Trace; c0106aeb <system_call+33/38>
Trace; c0110c5f <schedule_timeout+17/94>
Trace; c013b0f1 <do_select+19d/1dc>
Trace; c013b490 <sys_select+338/474>
Trace; c0106aeb <system_call+33/38>
Trace; c0110cbb <schedule_timeout+73/94>
Trace; c0110c00 <process_timeout+0/48>
Trace; c013b0f1 <do_select+19d/1dc>
Trace; c013b490 <sys_select+338/474>
Trace; c0106aeb <system_call+33/38>
Trace; c0105cc7 <sys_rt_sigsuspend+e3/100>
Trace; fffafefe <END_OF_CODE+2f6727b2/????>
Trace; c0106aeb <system_call+33/38>
Trace; c0110c5f <schedule_timeout+17/94>
Trace; c017754d <read_chan+3a9/6f0>
Trace; c0173738 <tty_read+b0/d0>
Trace; c012d7b6 <sys_read+96/cc>
Trace; c0106aeb <system_call+33/38>
Trace; c0110c5f <schedule_timeout+17/94>
Trace; c017754d <read_chan+3a9/6f0>
Trace; c0173738 <tty_read+b0/d0>
Trace; c012d7b6 <sys_read+96/cc>
Trace; c0106aeb <system_call+33/38>
Trace; c0105cc7 <sys_rt_sigsuspend+e3/100>
Trace; fffafefe <END_OF_CODE+2f6727b2/????>
Trace; c0106aeb <system_call+33/38>
Trace; c0105cc7 <sys_rt_sigsuspend+e3/100>
Trace; fffafefe <END_OF_CODE+2f6727b2/????>
Trace; c0106aeb <system_call+33/38>
Trace; c0110cbb <schedule_timeout+73/94>
Trace; c0110c00 <process_timeout+0/48>
Trace; c013b0f1 <do_select+19d/1dc>
Trace; c013b490 <sys_select+338/474>
Trace; c0106aeb <system_call+33/38>
Trace; c010002b <startup_32+2b/a5>
Trace; c0110c5f <schedule_timeout+17/94>
Trace; c013b0f1 <do_select+19d/1dc>
Trace; c013b490 <sys_select+338/474>
Trace; c0106aeb <system_call+33/38>
Trace; c0110c5f <schedule_timeout+17/94>
Trace; c017754d <read_chan+3a9/6f0>
Trace; c0173738 <tty_read+b0/d0>
Trace; c012d7b6 <sys_read+96/cc>
Trace; c0106aeb <system_call+33/38>
Trace; c0105cc7 <sys_rt_sigsuspend+e3/100>
Trace; fffafefe <END_OF_CODE+2f6727b2/????>
Trace; c0106aeb <system_call+33/38>
Trace; c013614c <pipe_wait+7c/a4>
Trace; c0136223 <pipe_read+af/1f8>
Trace; c012d7b6 <sys_read+96/cc>
Trace; c0106aeb <system_call+33/38>
Trace; c0110cbb <schedule_timeout+73/94>
Trace; c0110c00 <process_timeout+0/48>
Trace; c0119a48 <sys_nanosleep+108/180>
Trace; c0106aeb <system_call+33/38>
Trace; c012858c <__alloc_pages+104/240>
Trace; c011f0be <do_anonymous_page+32/90>
Trace; c011f14b <do_no_page+2f/e0>
Trace; c011f25b <handle_mm_fault+5f/cc>
Trace; c0110374 <do_page_fault+0/45c>
Trace; c01104d4 <do_page_fault+160/45c>
Trace; c0110374 <do_page_fault+0/45c>
Trace; c01aae1b <kfree_skbmem+b/54>
Trace; c01194f9 <update_process_times+1d/90>
Trace; c01193d7 <update_wall_time+b/3c>
Trace; c01195c4 <timer_bh+24/25c>
Trace; c010a9f8 <timer_interrupt+dc/158>
Trace; c0110f3c <schedule+250/370>
Trace; c0110374 <do_page_fault+0/45c>
Trace; c0106c18 <error_code+38/40>
Trace; c012858c <__alloc_pages+104/240>
Trace; c011eba2 <do_wp_page+166/244>
Trace; c011f291 <handle_mm_fault+95/cc>
Trace; c0110374 <do_page_fault+0/45c>
Trace; c01104d4 <do_page_fault+160/45c>
Trace; c0110374 <do_page_fault+0/45c>
Trace; c0128c75 <free_page_and_swap_cache+b5/b8>
Trace; c012d47c <filp_close+5c/64>
Trace; c0106c18 <error_code+38/40>
1 warning issued. Results may not be reliable.
5:03pm up 1:51, 4 users, load average: 0.66, 0.61, 0.25
procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
1 0 0 160528 79324 5708 165852 1 96 6 98 107 246 0 62 38
5:04pm up 1:52, 4 users, load average: 2.99, 1.26, 0.49
procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
3 0 0 160528 79664 5756 165852 1 95 6 97 107 250 0 62 37
