On 06/03/2014 05:34 PM, Tamas Papp wrote:
On 06/03/2014 05:08 PM, Stéphane Graber wrote:
On Tue, Jun 03, 2014 at 04:56:03PM +0200, Tamas Papp wrote:
On 06/03/2014 04:50 PM, Stéphane Graber wrote:
lxc-stop will send SIGPWR (or the equivalent signal) to the container,
wait 30s then SIGKILL init. lxc-stop -k will skip the SIGPWR step,
lxc-stop --nokill will skip the SIGKILL step.
It's pretty odd that init after a kill -9 is still marked
running... I'd
have expected it to either go away or get stuck in D state if
something's really wrong...
Do you see anything relevant in the kernel log?
Nothing. I was in hurry, so I restarted the whole machine, I cannot
collect more information.
Unfortunately I'm pretty sure it will be back soon, since this was
not the first time.
What do you suggest, what should I check, when I face it again?
So my hope would be for the kernel to report the task as hung which
causes a stacktrace to be dumped in dmesg. If not, then it's going to be
a bit harder to figure it out...
Is this valuable?
[514047.425278] ---[ end trace 3d2c1319330f8514 ]---
[514047.469086] kernel BUG at
/build/buildd/linux-3.13.0/mm/memory.c:3756!
[514047.490011] invalid opcode: 0000 [#10] SMP
[514047.510486] Modules linked in: joydev hid_generic usbhid hid
binfmt_misc veth vhost_net vhost macvtap macvlan xt_conntrack
ipt_REJECT ip6table_filter ip6_tables xt_CHECKSUM iptable_mangle
ipt_MASQUERADE iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4
nf_nat_ipv4 nf_nat nf_conntrack xt_tcpudp iptable_filter ip_tables
x_tables bridge stp llc gpio_ich x86_pkg_temp_thermal intel_powerclamp
coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul
ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper
ablk_helper cryptd serio_raw sb_edac edac_core lpc_ich hpwdt hpilo
ioatdma ipmi_si mac_hid acpi_power_meter lp parport zfs(POF)
zunicode(POF) zavl(POF) zcommon(POF) znvpair(POF) spl(OF) raid10
raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor
raid6_pq raid0 multipath linear igb i2c_algo_bit dca ahci ptp raid1
psmouse libahci pps_core hpsa
[514047.750609] CPU: 2 PID: 25087 Comm: java Tainted: PF D O
3.13.0-27-generic #50-Ubuntu
[514047.795328] Hardware name: HP ProLiant SL210t Gen8/, BIOS P83
12/20/2013
[514047.818632] task: ffff88175a5fc7d0 ti: ffff88176180e000 task.ti:
ffff88176180e000
[514047.865496] RIP: 0010:[<ffffffff811793d1>] [<ffffffff811793d1>]
handle_mm_fault+0xe61/0xf10
[514047.914325] RSP: 0018:ffff88176180fd98 EFLAGS: 00010246
[514047.939354] RAX: 0000000000000100 RBX: 00000007ff41a730 RCX:
ffff88176180fb10
[514047.989807] RDX: ffff88175a5fc7d0 RSI: 0000000000000000 RDI:
8000000cf2a009e6
[514048.040666] RBP: ffff88176180fe20 R08: 0000000000000000 R09:
00000000000000a9
[514048.092282] R10: 0000000000000001 R11: 0000000000000000 R12:
ffff881765ea5fd0
[514048.145498] R13: ffff88176ac77080 R14: ffff8802711ee200 R15:
0000000000000080
[514048.199709] FS: 00007f83947f7700(0000) GS:ffff88103fc40000(0000)
knlGS:0000000000000000
[514048.254465] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[514048.281837] CR2: 00007f6fdd6cec58 CR3: 0000000275364000 CR4:
00000000001427e0
[514048.335499] Stack:
[514048.361406] ffff88175a5fc7d0 0000000000000000 00007f839802f000
ffff8802711ee200
[514048.413528] 0000000000000f54 0000000000000000 0000000000000000
ffffea001e3c7000
[514048.465624] 800000078f1c0867 ffffea0043985670 ffffea00000000a9
ffff88176180fe00
[514048.517663] Call Trace:
[514048.542981] [<ffffffff81725924>] __do_page_fault+0x184/0x560
[514048.568584] [<ffffffff811112ec>] ? acct_account_cputime+0x1c/0x20
[514048.593936] [<ffffffff8109d76b>] ? account_user_time+0x8b/0xa0
[514048.618792] [<ffffffff8109dd84>] ? vtime_account_user+0x54/0x60
[514048.643188] [<ffffffff81725d1a>] do_page_fault+0x1a/0x70
[514048.667032] [<ffffffff81722188>] page_fault+0x28/0x30
[514048.690535] Code: ff 48 89 d9 4c 89 e2 4c 89 ee 4c 89 f7 44 89 4d
c8 e8 34 c1 ff ff 85 c0 0f 85 94 f5 ff ff 49 8b 3c 24 44 8b 4d c8 e9
68 f3 ff ff <0f> 0b be 8e 00 00 00 48 c7 c7 f8 39 a6 81 44 89 4d c8 e8
98 e3
[514048.763043] RIP [<ffffffff811793d1>] handle_mm_fault+0xe61/0xf10
[514048.786554] RSP <ffff88176180fd98>
[514048.809155] ------------[ cut here ]------------
[514048.809343] ---[ end trace 3d2c1319330f8515 ]---
[514048.856623] kernel BUG at
/build/buildd/linux-3.13.0/mm/memory.c:3756!
[514048.879824] invalid opcode: 0000 [#11] SMP
[514048.902099] Modules linked in: joydev hid_generic usbhid hid
binfmt_misc veth vhost_net vhost macvtap macvlan xt_conntrack
ipt_REJECT ip6table_filter ip6_tables xt_CHECKSUM iptable_mangle
ipt_MASQUERADE iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4
nf_nat_ipv4 nf_nat nf_conntrack xt_tcpudp iptable_filter ip_tables
x_tables bridge stp llc gpio_ich x86_pkg_temp_thermal intel_powerclamp
coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul
ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper
ablk_helper cryptd serio_raw sb_edac edac_core lpc_ich hpwdt hpilo
ioatdma ipmi_si mac_hid acpi_power_meter lp parport zfs(POF)
zunicode(POF) zavl(POF) zcommon(POF) znvpair(POF) spl(OF) raid10
raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor
raid6_pq raid0 multipath linear igb i2c_algo_bit dca ahci ptp raid1
psmouse libahci pps_core hpsa
[514049.154915] CPU: 16 PID: 25089 Comm: java Tainted: PF D O
3.13.0-27-generic #50-Ubuntu
[514049.200777] Hardware name: HP ProLiant SL210t Gen8/, BIOS P83
12/20/2013
[514049.224243] task: ffff88175a5fafe0 ti: ffff88175a724000 task.ti:
ffff88175a724000
[514049.271305] RIP: 0010:[<ffffffff811793d1>] [<ffffffff811793d1>]
handle_mm_fault+0xe61/0xf10
[514049.320379] RSP: 0000:ffff88175a725d98 EFLAGS: 00010246
[514049.345393] RAX: 0000000000000100 RBX: 00000007ff412730 RCX:
ffff88175a725b10
[514049.396064] RDX: ffff88175a5fafe0 RSI: 0000000000000000 RDI:
8000000cf2a009e6
[514049.446897] RBP: ffff88175a725e20 R08: 0000000000000000 R09:
00000000000000a9
[514049.498535] R10: 0000000000000001 R11: 0000000000000000 R12:
ffff881765ea5fd0
[514049.551984] R13: ffff88176ac77080 R14: ffff8802711ee200 R15:
0000000000000080
[514049.606194] FS: 00007f83945f5700(0000) GS:ffff88103fd40000(0000)
knlGS:0000000000000000
[514049.661165] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[514049.688513] CR2: 00000007ff412730 CR3: 0000000275364000 CR4:
00000000001427e0
[514049.742387] Stack:
[514049.768268] 0000000000000001 ffff88175a725db0 ffffffff8109a780
ffff88175a725dd0
[514049.820366] ffffffff810d7ad6 0000000000000001 ffffffff81f1f810
ffffea0025998400
[514049.872439] 8000000966610867 ffffea0043985670 ffffea00000000a9
00000001ffffffff
[514049.924437] Call Trace:
[514049.949730] [<ffffffff8109a780>] ? wake_up_state+0x10/0x20
[514049.975243] [<ffffffff810d7ad6>] ? wake_futex+0x66/0x90
[514050.000224] [<ffffffff81725924>] __do_page_fault+0x184/0x560
[514050.024857] [<ffffffff811112ec>] ? acct_account_cputime+0x1c/0x20
[514050.049169] [<ffffffff8109d76b>] ? account_user_time+0x8b/0xa0
[514050.072946] [<ffffffff8109dd84>] ? vtime_account_user+0x54/0x60
[514050.096475] [<ffffffff81725d1a>] do_page_fault+0x1a/0x70
[514050.119657] [<ffffffff81722188>] page_fault+0x28/0x30
[514050.142278] Code: ff 48 89 d9 4c 89 e2 4c 89 ee 4c 89 f7 44 89 4d
c8 e8 34 c1 ff ff 85 c0 0f 85 94 f5 ff ff 49 8b 3c 24 44 8b 4d c8 e9
68 f3 ff ff <0f> 0b be 8e 00 00 00 48 c7 c7 f8 39 a6 81 44 89 4d c8 e8
98 e3
[514050.211451] RIP [<ffffffff811793d1>] handle_mm_fault+0xe61/0xf10
[514050.233676] RSP <ffff88175a725d98>
[514050.255620] ---[ end trace 3d2c1319330f8516 ]---
[516197.062287] init: lxc-instance (fisheye1) main process (4489)
killed by KILL signal
Now it happened again, with a different container.
I see a lot of stuck processes, this time even a 'ps x' gets stuck:
1677 ? Ss 0:07 /lib/systemd/systemd-logind
1687 ? S 0:00 upstart-file-bridge --daemon
1695 ? Ss 0:44 /usr/bin/perl -wT /usr/sbin/munin-node
3022 ? S 0:00 [kworker/u130:5]
3035 tty4 Ss+ 0:00 /sbin/getty -8 38400 tty4
3039 tty5 Ss+ 0:00 /sbin/getty -8 38400 tty5
3052 tty2 Ss+ 0:00 /sbin/getty -8 38400 tty2
3058 tty3 Ss+ 0:00 /sbin/getty -8 38400 tty3
3064 tty6 Ss+ 0:00 /sbin/getty -8 38400 tty6
3066 ? S 0:00 [kworker/11:0]
3096 ? Ss 0:00 /usr/sbin/sshd -D
3112 ? Ss 0:01 cron
3115 ? Ss 0:00 sshd: tompos [priv]
3159 pts/0 S 0:00 sudo -i
3166 ? Ss 1:25 /usr/sbin/irqbalance
3189 pts/0 S+ 0:00 -bash
3200 ? Sl 0:00 /usr/sbin/libvirtd -d
3494 ? Ss 0:02 /usr/lib/postfix/master
3553 ? S< 0:00 [kworker/u132:2]
3566 ? Ss 0:00 lxc-start -n chemicalize
3699 ? Ss 0:02 [init]
4557 ? Ss 0:00 /sbin/mdadm --monitor --pid-file
/run/mdadm/monitor.pid --daemonise --scan --syslog
4629 tty1 Ss+ 0:00 /sbin/getty -8 38400 tty1
4693 ? D 0:00 pidof /usr/sbin/apache2
4865 ? S 0:00 [kauditd]
4889 ? S 0:07 [vhost-4888]
4892 ? S< 0:00 [zil_clean/0]
4895 ? S 0:00 [kvm-pit/4888]
4921 ? S 4:38 [vhost-4920]
4924 ? S< 0:00 [zil_clean/0]
4927 ? S 0:00 [kvm-pit/4920]
5092 ? S< 0:00 [kworker/u133:0]
5163 ? S 0:00 [kworker/u130:6]
5460 ? S 0:00 [kworker/5:2]
5466 ? S 0:00 [kworker/1:2]
5571 ? S< 0:00 [kworker/u132:1]
6007 ? D 0:00 /usr/bin/lsof -w -l +d /var/lib/php5
6010 ? Z 0:00 [lsof] <defunct>
6723 ? S 0:01 [kworker/10:2]
6725 ? S 0:00 [kworker/7:1]
6740 ? S 0:00 [kworker/15:2]
6750 ? S 0:00 [kworker/16:0]
7215 ? D 0:00 /usr/bin/lsof -w -l +d /var/lib/php5
7219 ? Z 0:00 [lsof] <defunct>
7938 ? D 0:00 /usr/bin/lsof -w -l +d /var/lib/php5
7942 ? Z 0:00 [lsof] <defunct>
7955 ? S 0:00 [kworker/u129:0]
8856 ? D 0:00 /usr/bin/lsof -w -l +d /var/lib/php5
8859 ? Z 0:00 [lsof] <defunct>
9477 ? D 0:00 /usr/bin/lsof -w -l +d /var/lib/php5
9480 ? Z 0:00 [lsof] <defunct>
10601 ? S 0:00 [kworker/3:0]
10750 ? D 0:00 /usr/bin/lsof -w -l +d /var/lib/php5
10753 ? Z 0:00 [lsof] <defunct>
11351 ? D 0:00 ps axf
11575 ? S 0:00 sudo -i
11599 ? S 0:00 -bash
11636 ? D 0:00 ps ax
12056 ? S 0:00 sudo -i
12079 ? S 0:00 -bash
12417 ? S 1:21 [kworker/9:0]
12735 ? S 0:00 [kworker/8:0]
13341 ? S 0:00 [kworker/12:2]
13367 ? S 0:00 [kworker/21:1]
13377 ? S 0:00 [kworker/4:0]
13651 ? S 0:00 [kworker/u129:1]
14237 ? S 7:03 [kworker/10:1]
15450 ? S 0:00 [kworker/6:0]
20520 ? S 0:00 [kworker/u130:0]
21166 ? S 0:00 [kworker/u129:3]
21324 ? S 1:57 [kworker/6:2]
21617 ? S 0:25 [kworker/21:2]
22163 ? S 0:00 [kworker/u129:4]
23529 ? D 0:00 pgrep -fa lxc-stop
23981 ? S 0:00 sudo -i
24026 ? S 0:00 -bash
24594 ? D 0:00 ps axf
24740 ? S 3:10 [kworker/16:1]
24796 ? S 0:00 sudo -i
24825 ? S 0:00 -bash
24869 ? D 0:00 ps ax
25155 ? Ss 0:00 sshd: tompos [priv]
25316 pts/11 S 0:00 sudo -i
25341 pts/11 S 0:00 -bash
25449 ? Ss 0:00 /usr/lib/x86_64-linux-gnu/lxc/lxc-monitord
/tank/lxc 5
Now I have a dmesg output, probably everything started, when this happened:
[423094.126247] ------------[ cut here ]------------
[423094.152443] kernel BUG at /build/buildd/linux-3.13.0/mm/memory.c:3756!
[423094.178742] invalid opcode: 0000 [#1] SMP
[423094.204554] Modules linked in: binfmt_misc vhost_net vhost macvtap
macvlan ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4
nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT xt_CHECKSUM
iptable_mangle xt_tcpudp veth ip6table_filter ip6_tables iptable_filter
ip_tables x_tables bridge stp llc x86_pkg_temp_thermal intel_powerclamp
coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel
gpio_ich aesni_intel aes_x86_64 lrw gf128mul sb_edac glue_helper
serio_raw lp hpwdt ablk_helper lpc_ich hpilo cryptd ioatdma edac_core
ipmi_si acpi_power_meter mac_hid parport zfs(POF) zunicode(POF)
zavl(POF) zcommon(POF) znvpair(POF) spl(OF) raid10 raid456
async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq
raid0 multipath linear igb i2c_algo_bit dca raid1 ahci ptp psmouse
libahci pps_core hpsa
[423094.495398] CPU: 13 PID: 11939 Comm: java Tainted: PF O
3.13.0-27-generic #50-Ubuntu
[423094.548747] Hardware name: HP ProLiant SL210t Gen8/, BIOS P83 12/20/2013
[423094.574988] task: ffff881ca7ec47d0 ti: ffff88163f05a000 task.ti:
ffff88163f05a000
[423094.627597] RIP: 0010:[<ffffffff811793d1>] [<ffffffff811793d1>]
handle_mm_fault+0xe61/0xf10
[423094.678564] RSP: 0000:ffff88163f05bd98 EFLAGS: 00010246
[423094.703767] RAX: 0000000000000100 RBX: 00000007d0200000 RCX:
ffff88163f05bb10
[423094.754679] RDX: ffff881ca7ec47d0 RSI: 0000000000000000 RDI:
8000000571c009e6
[423094.807019] RBP: ffff88163f05be20 R08: 0000000000000000 R09:
00000000000000a9
[423094.859710] R10: 0000000000000001 R11: 0000000000000000 R12:
ffff88133954a408
[423094.912560] R13: ffff880320a12cc0 R14: ffff881219b4d400 R15:
0000000000000080
[423094.966237] FS: 00007fc5f1825700(0000) GS:ffff88103fce0000(0000)
knlGS:0000000000000000
[423095.020459] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[423095.047460] CR2: 00000007d0200000 CR3: 00000017cd0d9000 CR4:
00000000001427e0
[423095.100677] Stack:
[423095.126512] 0000000000000001 ffff88163f05bdb0 ffffffff8109a780
ffff88163f05bdd0
[423095.177861] ffffffff810d7ad6 0000000000000001 ffffffff81f1ebb0
ffff88163f05be78
[423095.229467] ffffffff810d983d ffff88163f05be48 00000000000000a9
00000001ffffffff
[423095.280978] Call Trace:
[423095.306138] [<ffffffff8109a780>] ? wake_up_state+0x10/0x20
[423095.331531] [<ffffffff810d7ad6>] ? wake_futex+0x66/0x90
[423095.356962] [<ffffffff810d983d>] ? futex_wake_op+0x4ed/0x620
[423095.381663] [<ffffffff81725924>] __do_page_fault+0x184/0x560
[423095.405532] [<ffffffff811112ec>] ? acct_account_cputime+0x1c/0x20
[423095.429089] [<ffffffff8109d76b>] ? account_user_time+0x8b/0xa0
[423095.452361] [<ffffffff8109dd84>] ? vtime_account_user+0x54/0x60
[423095.475382] [<ffffffff81725d1a>] do_page_fault+0x1a/0x70
[423095.498153] [<ffffffff81722188>] page_fault+0x28/0x30
[423095.520058] Code: ff 48 89 d9 4c 89 e2 4c 89 ee 4c 89 f7 44 89 4d c8
e8 34 c1 ff ff 85 c0 0f 85 94 f5 ff ff 49 8b 3c 24 44 8b 4d c8 e9 68 f3
ff ff <0f> 0b be 8e 00 00 00 48 c7 c7 f8 39 a6 81 44 89 4d c8 e8 98 e3
[423095.586301] RIP [<ffffffff811793d1>] handle_mm_fault+0xe61/0xf10
[423095.607798] RSP <ffff88163f05bd98>
[423095.659448] ---[ end trace a2f5bece5536224a ]---
But I'm not sure, in syslog I can see only this:
Jun 8 13:44:17 v306 kernel: [423094.912560] R13: ffff880320a12cc0 R14:
ffff881219b4d400 R15: 0000000000000080
Jun 8 13:44:17 v306 kernel: [423095.498153] [<ffffffff81722188>]
page_fault+0x28/0x30
But I can see puppet activity in syslog after this point, so I'm not
sure at all.
$ echo m /proc/sysrq-trigger
[436815.646569] SysRq : Show Memory
[436815.671215] Mem-Info:
[436815.695291] Node 0 DMA per-cpu:
[436815.720109] CPU 0: hi: 0, btch: 1 usd: 0
[436815.744586] CPU 1: hi: 0, btch: 1 usd: 0
[436815.768253] CPU 2: hi: 0, btch: 1 usd: 0
[436815.791281] CPU 3: hi: 0, btch: 1 usd: 0
[436815.813492] CPU 4: hi: 0, btch: 1 usd: 0
[436815.834925] CPU 5: hi: 0, btch: 1 usd: 0
[436815.855547] CPU 6: hi: 0, btch: 1 usd: 0
[436815.875366] CPU 7: hi: 0, btch: 1 usd: 0
[436815.895133] CPU 8: hi: 0, btch: 1 usd: 0
[436815.914074] CPU 9: hi: 0, btch: 1 usd: 0
[436815.932275] CPU 10: hi: 0, btch: 1 usd: 0
[436815.949823] CPU 11: hi: 0, btch: 1 usd: 0
[436815.967354] CPU 12: hi: 0, btch: 1 usd: 0
[436815.984089] CPU 13: hi: 0, btch: 1 usd: 0
[436816.000033] CPU 14: hi: 0, btch: 1 usd: 0
[436816.015874] CPU 15: hi: 0, btch: 1 usd: 0
[436816.031694] CPU 16: hi: 0, btch: 1 usd: 0
[436816.046706] CPU 17: hi: 0, btch: 1 usd: 0
[436816.060922] CPU 18: hi: 0, btch: 1 usd: 0
[436816.074508] CPU 19: hi: 0, btch: 1 usd: 0
[436816.087294] CPU 20: hi: 0, btch: 1 usd: 0
[436816.099308] CPU 21: hi: 0, btch: 1 usd: 0
[436816.110240] CPU 22: hi: 0, btch: 1 usd: 0
[436816.120043] CPU 23: hi: 0, btch: 1 usd: 0
[436816.128585] Node 0 DMA32 per-cpu:
[436816.136225] CPU 0: hi: 186, btch: 31 usd: 37
[436816.143444] CPU 1: hi: 186, btch: 31 usd: 174
[436816.149946] CPU 2: hi: 186, btch: 31 usd: 100
[436816.156165] CPU 3: hi: 186, btch: 31 usd: 53
[436816.162173] CPU 4: hi: 186, btch: 31 usd: 109
[436816.167750] CPU 5: hi: 186, btch: 31 usd: 173
[436816.173344] CPU 6: hi: 186, btch: 31 usd: 0
[436816.178987] CPU 7: hi: 186, btch: 31 usd: 0
[436816.184628] CPU 8: hi: 186, btch: 31 usd: 0
[436816.190250] CPU 9: hi: 186, btch: 31 usd: 0
[436816.195878] CPU 10: hi: 186, btch: 31 usd: 0
[436816.201506] CPU 11: hi: 186, btch: 31 usd: 0
[436816.207166] CPU 12: hi: 186, btch: 31 usd: 79
[436816.212843] CPU 13: hi: 186, btch: 31 usd: 164
[436816.218555] CPU 14: hi: 186, btch: 31 usd: 165
[436816.224265] CPU 15: hi: 186, btch: 31 usd: 156
[436816.229992] CPU 16: hi: 186, btch: 31 usd: 125
[436816.235737] CPU 17: hi: 186, btch: 31 usd: 71
[436816.241529] CPU 18: hi: 186, btch: 31 usd: 0
[436816.247343] CPU 19: hi: 186, btch: 31 usd: 0
[436816.253138] CPU 20: hi: 186, btch: 31 usd: 0
[436816.258967] CPU 21: hi: 186, btch: 31 usd: 0
[436816.264778] CPU 22: hi: 186, btch: 31 usd: 0
[436816.270588] CPU 23: hi: 186, btch: 31 usd: 0
[436816.276379] Node 0 Normal per-cpu:
[436816.282375] CPU 0: hi: 186, btch: 31 usd: 129
[436816.289083] CPU 1: hi: 186, btch: 31 usd: 77
[436816.295807] CPU 2: hi: 186, btch: 31 usd: 175
[436816.302594] CPU 3: hi: 186, btch: 31 usd: 64
[436816.308984] CPU 4: hi: 186, btch: 31 usd: 111
[436816.314912] CPU 5: hi: 186, btch: 31 usd: 161
[436816.320789] CPU 6: hi: 186, btch: 31 usd: 0
[436816.326713] CPU 7: hi: 186, btch: 31 usd: 0
[436816.332553] CPU 8: hi: 186, btch: 31 usd: 0
[436816.338343] CPU 9: hi: 186, btch: 31 usd: 0
[436816.344081] CPU 10: hi: 186, btch: 31 usd: 0
[436816.349794] CPU 11: hi: 186, btch: 31 usd: 0
[436816.355504] CPU 12: hi: 186, btch: 31 usd: 86
[436816.361230] CPU 13: hi: 186, btch: 31 usd: 128
[436816.367007] CPU 14: hi: 186, btch: 31 usd: 154
[436816.372806] CPU 15: hi: 186, btch: 31 usd: 74
[436816.378602] CPU 16: hi: 186, btch: 31 usd: 180
[436816.384414] CPU 17: hi: 186, btch: 31 usd: 168
[436816.390208] CPU 18: hi: 186, btch: 31 usd: 0
[436816.396054] CPU 19: hi: 186, btch: 31 usd: 25
[436816.401902] CPU 20: hi: 186, btch: 31 usd: 0
[436816.407797] CPU 21: hi: 186, btch: 31 usd: 0
[436816.413628] CPU 22: hi: 186, btch: 31 usd: 0
[436816.419424] CPU 23: hi: 186, btch: 31 usd: 0
[436816.425204] Node 1 Normal per-cpu:
[436816.431188] CPU 0: hi: 186, btch: 31 usd: 0
[436816.437898] CPU 1: hi: 186, btch: 31 usd: 0
[436816.444571] CPU 2: hi: 186, btch: 31 usd: 0
[436816.451260] CPU 3: hi: 186, btch: 31 usd: 0
[436816.457486] CPU 4: hi: 186, btch: 31 usd: 155
[436816.463282] CPU 5: hi: 186, btch: 31 usd: 56
[436816.469060] CPU 6: hi: 186, btch: 31 usd: 59
[436816.474803] CPU 7: hi: 186, btch: 31 usd: 55
[436816.480493] CPU 8: hi: 186, btch: 31 usd: 178
[436816.486202] CPU 9: hi: 186, btch: 31 usd: 115
[436816.491926] CPU 10: hi: 186, btch: 31 usd: 29
[436816.497670] CPU 11: hi: 186, btch: 31 usd: 39
[436816.503446] CPU 12: hi: 186, btch: 31 usd: 0
[436816.509287] CPU 13: hi: 186, btch: 31 usd: 18
[436816.515183] CPU 14: hi: 186, btch: 31 usd: 74
[436816.521080] CPU 15: hi: 186, btch: 31 usd: 0
[436816.527007] CPU 16: hi: 186, btch: 31 usd: 0
[436816.532882] CPU 17: hi: 186, btch: 31 usd: 0
[436816.538709] CPU 18: hi: 186, btch: 31 usd: 135
[436816.544554] CPU 19: hi: 186, btch: 31 usd: 164
[436816.550421] CPU 20: hi: 186, btch: 31 usd: 167
[436816.556303] CPU 21: hi: 186, btch: 31 usd: 123
[436816.562164] CPU 22: hi: 186, btch: 31 usd: 116
[436816.568012] CPU 23: hi: 186, btch: 31 usd: 50
[436816.573882] active_anon:2919029 inactive_anon:27295 isolated_anon:0
[436816.573882] active_file:801196 inactive_file:1948997 isolated_file:0
[436816.573882] unevictable:0 dirty:0 writeback:0 unstable:0
[436816.573882] free:12852299 slab_reclaimable:283058
slab_unreclaimable:276326
[436816.573882] mapped:15018 shmem:557 pagetables:8924 bounce:0
[436816.573882] free_cma:0
[436816.624593] Node 0 DMA free:15828kB min:8kB low:8kB high:12kB
active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB
unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15976kB
managed:15892kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB
slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB
pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB
pages_scanned:0 all_unreclaimable? yes
[436816.707576] lowmem_reserve[]: 0 1789 64251 64251
[436816.724738] Node 0 DMA32 free:1076484kB min:1252kB low:1564kB
high:1876kB active_anon:136364kB inactive_anon:16360kB
active_file:20572kB inactive_file:50928kB unevictable:0kB
isolated(anon):0kB isolated(file):0kB present:1914544kB
managed:1835596kB mlocked:0kB dirty:0kB writeback:0kB mapped:456kB
shmem:28kB slab_reclaimable:13220kB slab_unreclaimable:11172kB
kernel_stack:40kB pagetables:796kB unstable:0kB bounce:0kB free_cma:0kB
writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[436816.876588] lowmem_reserve[]: 0 0 62462 62462
[436816.902021] Node 0 Normal free:24898544kB min:43708kB low:54632kB
high:65560kB active_anon:6981544kB inactive_anon:90080kB
active_file:1865844kB inactive_file:5616424kB unevictable:0kB
isolated(anon):0kB isolated(file):0kB present:65011712kB
managed:63961116kB mlocked:16kB dirty:20kB writeback:0kB mapped:17608kB
shmem:1472kB slab_reclaimable:429048kB slab_unreclaimable:418516kB
kernel_stack:4760kB pagetables:20188kB unstable:0kB bounce:0kB
free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[436817.114072] lowmem_reserve[]: 0 0 0 0
[436817.147814] Node 1 Normal free:25418340kB min:45140kB low:56424kB
high:67708kB active_anon:4558208kB inactive_anon:2740kB
active_file:1318368kB inactive_file:2128636kB unevictable:0kB
isolated(anon):0kB isolated(file):0kB present:67108860kB
managed:66055976kB mlocked:16kB dirty:0kB writeback:0kB mapped:42008kB
shmem:728kB slab_reclaimable:689964kB slab_unreclaimable:675616kB
kernel_stack:2640kB pagetables:14760kB unstable:0kB bounce:0kB
free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[436817.395923] lowmem_reserve[]: 0 0 0 0
[436817.431221] Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 1*64kB (U)
1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) =
15828kB
[436817.502378] Node 0 DMA32: 4749*4kB (UEM) 4478*8kB (UEM) 2182*16kB
(UEM) 1562*32kB (UEM) 965*64kB (UEM) 516*128kB (UM) 234*256kB (UM)
97*512kB (UM) 71*1024kB (UEM) 118*2048kB (MR) 94*4096kB (M) = 1076484kB
[436817.608847] Node 0 Normal: 297772*4kB (UEM) 105932*8kB (UEM)
30526*16kB (UEM) 11629*32kB (UEM) 5556*64kB (UEM) 3070*128kB (UM)
2380*256kB (UEM) 1798*512kB (UEM) 1875*1024kB (UM) 3684*2048kB (MR)
2504*4096kB (M) = 24898704kB
[436817.717184] Node 1 Normal: 234109*4kB (UEM) 129663*8kB (UEM)
59754*16kB (UEM) 18947*32kB (UEM) 6832*64kB (UEM) 5715*128kB (UEM)
4586*256kB (UEM) 4855*512kB (UEM) 3978*1024kB (UEM) 2618*2048kB (M)
1860*4096kB (MR) = 25418348kB
[436817.825583] Node 0 hugepages_total=0 hugepages_free=0
hugepages_surp=0 hugepages_size=2048kB
[436817.897389] Node 1 hugepages_total=0 hugepages_free=0
hugepages_surp=0 hugepages_size=2048kB
[436817.969947] 2756671 total pagecache pages
[436818.005541] 5919 pages in swap cache
[436818.040308] Swap cache stats: add 91884, delete 85965, find 87132/88084
[436818.075267] Free swap = 19451316kB
[436818.109350] Total swap = 19514748kB
[436818.142606] 33512773 pages RAM
[436818.175612] 0 pages HighMem/MovableOnly
[436818.208207] 263221 pages reserved
After 'echo t' the machine got instable and hanged up completely.
Can it be HW issue?
tamas
_______________________________________________
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users
