I just had this occur today on a KVM guest running Ubuntu Server 10.04.1 64-bit. During a period of heavy network traffic, networking stopped working and I was unable to log into the virtual machine remotely. I was forced to kill and restart it using virsh on the KVM host machine (also running Ubuntu Server 10.04.1 64-bit).
My kernel is: 2.6.32-26-server #47-Ubuntu SMP Wed Nov 17 17:05:29 UTC 2010 x86_64 GNU/Linux Here's what appeared in my syslog at the moment of the failure: Nov 29 17:51:01 virtuala2 kernel: [316382.936157] kswapd0: page allocation failure. order:0, mode:0x4020 Nov 29 17:51:01 virtuala2 kernel: [316382.936162] Pid: 36, comm: kswapd0 Not tainted 2.6.32-26-server #47-Ubuntu Nov 29 17:51:01 virtuala2 kernel: [316382.936165] Call Trace: Nov 29 17:51:01 virtuala2 kernel: [316382.936167] <IRQ> [<ffffffff810f9a69>] __alloc_pages_slowpath+0x4a9/0x590 Nov 29 17:51:01 virtuala2 kernel: [316382.936178] [<ffffffff810f9cc1>] __alloc_pages_nodemask+0x171/0x180 Nov 29 17:51:01 virtuala2 kernel: [316382.936184] [<ffffffff8112ccf7>] alloc_pages_current+0x87/0xd0 Nov 29 17:51:01 virtuala2 kernel: [316382.936188] [<ffffffff81132d37>] new_slab+0x2f7/0x310 Nov 29 17:51:01 virtuala2 kernel: [316382.936191] [<ffffffff811355e1>] __slab_alloc+0x201/0x2d0 Nov 29 17:51:01 virtuala2 kernel: [316382.936196] [<ffffffff81469796>] ? __netdev_alloc_skb+0x36/0x60 Nov 29 17:51:01 virtuala2 kernel: [316382.936200] [<ffffffff811365bf>] __kmalloc_node_track_caller+0xaf/0x160 Nov 29 17:51:01 virtuala2 kernel: [316382.936204] [<ffffffff81469796>] ? __netdev_alloc_skb+0x36/0x60 Nov 29 17:51:01 virtuala2 kernel: [316382.936207] [<ffffffff81468fb0>] __alloc_skb+0x80/0x190 Nov 29 17:51:01 virtuala2 kernel: [316382.936210] [<ffffffff81469796>] __netdev_alloc_skb+0x36/0x60 Nov 29 17:51:01 virtuala2 kernel: [316382.936215] [<ffffffff813dacc4>] try_fill_recv+0x54/0x200 Nov 29 17:51:01 virtuala2 kernel: [316382.936218] [<ffffffff813db01d>] virtnet_poll+0x10d/0x160 Nov 29 17:51:01 virtuala2 kernel: [316382.936222] [<ffffffff8147356f>] net_rx_action+0x10f/0x250 Nov 29 17:51:01 virtuala2 kernel: [316382.936226] [<ffffffff8106d487>] __do_softirq+0xb7/0x1e0 Nov 29 17:51:01 virtuala2 kernel: [316382.936230] [<ffffffff810c4060>] ? handle_IRQ_event+0x60/0x170 Nov 29 17:51:01 virtuala2 kernel: [316382.936233] [<ffffffff8106d4e5>] ? __do_softirq+0x115/0x1e0 Nov 29 17:51:01 virtuala2 kernel: [316382.936237] [<ffffffff810132ec>] call_softirq+0x1c/0x30 Nov 29 17:51:01 virtuala2 kernel: [316382.936240] [<ffffffff81014cb5>] do_softirq+0x65/0xa0 Nov 29 17:51:01 virtuala2 kernel: [316382.936243] [<ffffffff8106d325>] irq_exit+0x85/0x90 Nov 29 17:51:01 virtuala2 kernel: [316382.936248] [<ffffffff8155f905>] do_IRQ+0x75/0xf0 Nov 29 17:51:01 virtuala2 kernel: [316382.936272] [<ffffffff81012b13>] ret_from_intr+0x0/0x11 Nov 29 17:51:01 virtuala2 kernel: [316382.936274] <EOI> [<ffffffff810fb1ac>] ? free_hot_cold_page+0xdc/0x240 Nov 29 17:51:01 virtuala2 kernel: [316382.936297] [<ffffffff8121ab47>] ? jbd2_journal_try_to_free_buffers+0xa7/0x150 Nov 29 17:51:01 virtuala2 kernel: [316382.936302] [<ffffffff810fb369>] ? __pagevec_free+0x59/0xb0 Nov 29 17:51:01 virtuala2 kernel: [316382.936306] [<ffffffff8113dd30>] ? mem_cgroup_uncharge_cache_page+0x10/0x20 Nov 29 17:51:01 virtuala2 kernel: [316382.936310] [<ffffffff81101795>] ? shrink_page_list+0x415/0x600 Nov 29 17:51:01 virtuala2 kernel: [316382.936314] [<ffffffff81101c8e>] ? shrink_inactive_list+0x30e/0x810 Nov 29 17:51:01 virtuala2 kernel: [316382.936319] [<ffffffff810fbe7a>] ? determine_dirtyable_memory+0x1a/0x30 Nov 29 17:51:01 virtuala2 kernel: [316382.936323] [<ffffffff810fbf27>] ? get_dirty_limits+0x27/0x2f0 Nov 29 17:51:01 virtuala2 kernel: [316382.936326] [<ffffffff81102221>] ? shrink_list+0x91/0xf0 Nov 29 17:51:01 virtuala2 kernel: [316382.936329] [<ffffffff81102417>] ? shrink_zone+0x197/0x240 Nov 29 17:51:01 virtuala2 kernel: [316382.936332] [<ffffffff811034a9>] ? balance_pgdat+0x659/0x6d0 Nov 29 17:51:01 virtuala2 kernel: [316382.936336] [<ffffffff811004d0>] ? isolate_pages_global+0x0/0x50 Nov 29 17:51:01 virtuala2 kernel: [316382.936339] [<ffffffff8110361e>] ? kswapd+0xfe/0x150 Nov 29 17:51:01 virtuala2 kernel: [316382.936343] [<ffffffff81084250>] ? autoremove_wake_function+0x0/0x40 Nov 29 17:51:01 virtuala2 kernel: [316382.936346] [<ffffffff81103520>] ? kswapd+0x0/0x150 Nov 29 17:51:01 virtuala2 kernel: [316382.936349] [<ffffffff81083ed6>] ? kthread+0x96/0xa0 Nov 29 17:51:01 virtuala2 kernel: [316382.936353] [<ffffffff810131ea>] ? child_rip+0xa/0x20 Nov 29 17:51:01 virtuala2 kernel: [316382.936356] [<ffffffff81083e40>] ? kthread+0x0/0xa0 Nov 29 17:51:01 virtuala2 kernel: [316382.936358] [<ffffffff810131e0>] ? child_rip+0x0/0x20 Nov 29 17:51:01 virtuala2 kernel: [316382.936360] Mem-Info: Nov 29 17:51:01 virtuala2 kernel: [316382.936362] Node 0 DMA per-cpu: Nov 29 17:51:01 virtuala2 kernel: [316382.936365] CPU 0: hi: 0, btch: 1 usd: 0 Nov 29 17:51:01 virtuala2 kernel: [316382.936367] CPU 1: hi: 0, btch: 1 usd: 0 Nov 29 17:51:01 virtuala2 kernel: [316382.936369] Node 0 DMA32 per-cpu: Nov 29 17:51:01 virtuala2 kernel: [316382.936371] CPU 0: hi: 186, btch: 31 usd: 197 Nov 29 17:51:01 virtuala2 kernel: [316382.936374] CPU 1: hi: 186, btch: 31 usd: 2 Nov 29 17:51:01 virtuala2 kernel: [316382.936378] active_anon:68103 inactive_anon:27482 isolated_anon:0 Nov 29 17:51:01 virtuala2 kernel: [316382.936380] active_file:183026 inactive_file:195079 isolated_file:33 Nov 29 17:51:01 virtuala2 kernel: [316382.936381] unevictable:0 dirty:43777 writeback:527 unstable:0 Nov 29 17:51:01 virtuala2 kernel: [316382.936382] free:2511 slab_reclaimable:19556 slab_unreclaimable:2687 Nov 29 17:51:01 virtuala2 kernel: [316382.936383] mapped:2650 shmem:76 pagetables:2145 bounce:0 Nov 29 17:51:01 virtuala2 kernel: [316382.936385] Node 0 DMA free:8020kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:536kB inactive_file:7100kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15344kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:228kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Nov 29 17:51:01 virtuala2 kernel: [316382.936395] lowmem_reserve[]: 0 2004 2004 2004 Nov 29 17:51:01 virtuala2 kernel: [316382.936399] Node 0 DMA32 free:2024kB min:5704kB low:7128kB high:8556kB active_anon:272412kB inactive_anon:109928kB active_file:731568kB inactive_file:773216kB unevictable:0kB isolated(anon):0kB isolated(file):132kB present:2052300kB mlocked:0kB dirty:175108kB writeback:2108kB mapped:10600kB shmem:304kB slab_reclaimable:77996kB slab_unreclaimable:10732kB kernel_stack:1184kB pagetables:8580kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:197 all_unreclaimable? no Nov 29 17:51:01 virtuala2 kernel: [316382.936411] lowmem_reserve[]: 0 0 0 0 Nov 29 17:51:01 virtuala2 kernel: [316382.936414] Node 0 DMA: 5*4kB 2*8kB 5*16kB 3*32kB 2*64kB 4*128kB 4*256kB 4*512kB 2*1024kB 1*2048kB 0*4096kB = 8020kB Nov 29 17:51:01 virtuala2 kernel: [316382.936423] Node 0 DMA32: 166*4kB 94*8kB 0*16kB 1*32kB 1*64kB 0*128kB 0*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 2024kB Nov 29 17:51:01 virtuala2 kernel: [316382.936432] 378180 total pagecache pages Nov 29 17:51:01 virtuala2 kernel: [316382.936434] 0 pages in swap cache Nov 29 17:51:01 virtuala2 kernel: [316382.936436] Swap cache stats: add 0, delete 0, find 0/0 Nov 29 17:51:01 virtuala2 kernel: [316382.936438] Free swap = 3444728kB Nov 29 17:51:01 virtuala2 kernel: [316382.936439] Total swap = 3444728kB Nov 29 17:51:01 virtuala2 kernel: [316382.943016] 524283 pages RAM Nov 29 17:51:01 virtuala2 kernel: [316382.943018] 9848 pages reserved Nov 29 17:51:01 virtuala2 kernel: [316382.943020] 400801 pages shared Nov 29 17:51:01 virtuala2 kernel: [316382.943021] 135727 pages non-shared Nov 29 17:51:01 virtuala2 kernel: [316382.943025] SLUB: Unable to allocate memory on node -1 (gfp=0x20) Nov 29 17:51:01 virtuala2 kernel: [316382.943029] cache: kmalloc-1024, object size: 1024, buffer size: 1024, default order: 2, min order: 0 Nov 29 17:51:01 virtuala2 kernel: [316382.943032] node 0: slabs: 149, objs: 2048, free: 0 Nov 29 17:51:01 virtuala2 kernel: [316382.943344] kswapd0: page allocation failure. order:0, mode:0x20 Nov 29 17:51:01 virtuala2 kernel: [316382.943369] Pid: 36, comm: kswapd0 Not tainted 2.6.32-26-server #47-Ubuntu Nov 29 17:51:01 virtuala2 kernel: [316382.943372] Call Trace: Nov 29 17:51:01 virtuala2 kernel: [316382.943374] <IRQ> [<ffffffff810f9a69>] __alloc_pages_slowpath+0x4a9/0x590 Nov 29 17:51:01 virtuala2 kernel: [316382.943386] [<ffffffff810f9cc1>] __alloc_pages_nodemask+0x171/0x180 Nov 29 17:51:01 virtuala2 kernel: [316382.943392] [<ffffffff8112ccf7>] alloc_pages_current+0x87/0xd0 Nov 29 17:51:01 virtuala2 kernel: [316382.943398] [<ffffffff813dadf2>] try_fill_recv+0x182/0x200 Nov 29 17:51:01 virtuala2 kernel: [316382.943402] [<ffffffff813db01d>] virtnet_poll+0x10d/0x160 Nov 29 17:51:01 virtuala2 kernel: [316382.943406] [<ffffffff8147356f>] net_rx_action+0x10f/0x250 Nov 29 17:51:01 virtuala2 kernel: [316382.943411] [<ffffffff8106d487>] __do_softirq+0xb7/0x1e0 Nov 29 17:51:01 virtuala2 kernel: [316382.943416] [<ffffffff810c4060>] ? handle_IRQ_event+0x60/0x170 Nov 29 17:51:01 virtuala2 kernel: [316382.943419] [<ffffffff8106d4e5>] ? __do_softirq+0x115/0x1e0 Nov 29 17:51:01 virtuala2 kernel: [316382.943423] [<ffffffff810132ec>] call_softirq+0x1c/0x30 Nov 29 17:51:01 virtuala2 kernel: [316382.943427] [<ffffffff81014cb5>] do_softirq+0x65/0xa0 Nov 29 17:51:01 virtuala2 kernel: [316382.943430] [<ffffffff8106d325>] irq_exit+0x85/0x90 Nov 29 17:51:01 virtuala2 kernel: [316382.943435] [<ffffffff8155f905>] do_IRQ+0x75/0xf0 Nov 29 17:51:01 virtuala2 kernel: [316382.943438] [<ffffffff81012b13>] ret_from_intr+0x0/0x11 Nov 29 17:51:01 virtuala2 kernel: [316382.943440] <EOI> [<ffffffff810fb1ac>] ? free_hot_cold_page+0xdc/0x240 Nov 29 17:51:01 virtuala2 kernel: [316382.943449] [<ffffffff8121ab47>] ? jbd2_journal_try_to_free_buffers+0xa7/0x150 Nov 29 17:51:01 virtuala2 kernel: [316382.943453] [<ffffffff810fb369>] ? __pagevec_free+0x59/0xb0 Nov 29 17:51:01 virtuala2 kernel: [316382.943457] [<ffffffff8113dd30>] ? mem_cgroup_uncharge_cache_page+0x10/0x20 Nov 29 17:51:01 virtuala2 kernel: [316382.943461] [<ffffffff81101795>] ? shrink_page_list+0x415/0x600 Nov 29 17:51:01 virtuala2 kernel: [316382.943465] [<ffffffff81101c8e>] ? shrink_inactive_list+0x30e/0x810 Nov 29 17:51:01 virtuala2 kernel: [316382.943474] [<ffffffff810fbe7a>] ? determine_dirtyable_memory+0x1a/0x30 Nov 29 17:51:01 virtuala2 kernel: [316382.943477] [<ffffffff810fbf27>] ? get_dirty_limits+0x27/0x2f0 Nov 29 17:51:01 virtuala2 kernel: [316382.943480] [<ffffffff81102221>] ? shrink_list+0x91/0xf0 Nov 29 17:51:01 virtuala2 kernel: [316382.943483] [<ffffffff81102417>] ? shrink_zone+0x197/0x240 Nov 29 17:51:01 virtuala2 kernel: [316382.943486] [<ffffffff811034a9>] ? balance_pgdat+0x659/0x6d0 Nov 29 17:51:01 virtuala2 kernel: [316382.943490] [<ffffffff811004d0>] ? isolate_pages_global+0x0/0x50 Nov 29 17:51:01 virtuala2 kernel: [316382.943493] [<ffffffff8110361e>] ? kswapd+0xfe/0x150 Nov 29 17:51:01 virtuala2 kernel: [316382.943497] [<ffffffff81084250>] ? autoremove_wake_function+0x0/0x40 Nov 29 17:51:01 virtuala2 kernel: [316382.943501] [<ffffffff81103520>] ? kswapd+0x0/0x150 Nov 29 17:51:01 virtuala2 kernel: [316382.943503] [<ffffffff81083ed6>] ? kthread+0x96/0xa0 Nov 29 17:51:01 virtuala2 kernel: [316382.943507] [<ffffffff810131ea>] ? child_rip+0xa/0x20 Nov 29 17:51:01 virtuala2 kernel: [316382.943510] [<ffffffff81083e40>] ? kthread+0x0/0xa0 Nov 29 17:51:01 virtuala2 kernel: [316382.943513] [<ffffffff810131e0>] ? child_rip+0x0/0x20 Nov 29 17:51:01 virtuala2 kernel: [316382.943514] Mem-Info: Nov 29 17:51:01 virtuala2 kernel: [316382.943516] Node 0 DMA per-cpu: Nov 29 17:51:01 virtuala2 kernel: [316382.943519] CPU 0: hi: 0, btch: 1 usd: 0 Nov 29 17:51:01 virtuala2 kernel: [316382.943521] CPU 1: hi: 0, btch: 1 usd: 0 Nov 29 17:51:01 virtuala2 kernel: [316382.943523] Node 0 DMA32 per-cpu: Nov 29 17:51:01 virtuala2 kernel: [316382.943525] CPU 0: hi: 186, btch: 31 usd: 30 Nov 29 17:51:01 virtuala2 kernel: [316382.943527] CPU 1: hi: 186, btch: 31 usd: 0 Nov 29 17:51:01 virtuala2 kernel: [316382.943532] active_anon:68103 inactive_anon:27482 isolated_anon:0 Nov 29 17:51:01 virtuala2 kernel: [316382.943533] active_file:183026 inactive_file:194951 isolated_file:33 Nov 29 17:51:01 virtuala2 kernel: [316382.943534] unevictable:0 dirty:43777 writeback:527 unstable:0 Nov 29 17:51:01 virtuala2 kernel: [316382.943536] free:2596 slab_reclaimable:19556 slab_unreclaimable:2750 Nov 29 17:51:01 virtuala2 kernel: [316382.943537] mapped:2650 shmem:76 pagetables:2145 bounce:0 Nov 29 17:51:01 virtuala2 kernel: [316382.943539] Node 0 DMA free:8020kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:536kB inactive_file:7100kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15344kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:228kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Nov 29 17:51:01 virtuala2 kernel: [316382.943550] lowmem_reserve[]: 0 2004 2004 2004 Nov 29 17:51:01 virtuala2 kernel: [316382.943554] Node 0 DMA32 free:2364kB min:5704kB low:7128kB high:8556kB active_anon:272412kB inactive_anon:109928kB active_file:731568kB inactive_file:772704kB unevictable:0kB isolated(anon):0kB isolated(file):132kB present:2052300kB mlocked:0kB dirty:175108kB writeback:2108kB mapped:10600kB shmem:304kB slab_reclaimable:77996kB slab_unreclaimable:10984kB kernel_stack:1184kB pagetables:8580kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Nov 29 17:51:01 virtuala2 kernel: [316382.943565] lowmem_reserve[]: 0 0 0 0 Nov 29 17:51:01 virtuala2 kernel: [316382.943578] Node 0 DMA: 5*4kB 2*8kB 5*16kB 3*32kB 2*64kB 4*128kB 4*256kB 4*512kB 2*1024kB 1*2048kB 0*4096kB = 8020kB Nov 29 17:51:01 virtuala2 kernel: [316382.943588] Node 0 DMA32: 183*4kB 138*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 2364kB Nov 29 17:51:01 virtuala2 kernel: [316382.943598] 378056 total pagecache pages Nov 29 17:51:01 virtuala2 kernel: [316382.943599] 0 pages in swap cache Nov 29 17:51:01 virtuala2 kernel: [316382.943601] Swap cache stats: add 0, delete 0, find 0/0 Nov 29 17:51:01 virtuala2 kernel: [316382.943603] Free swap = 3444728kB Nov 29 17:51:01 virtuala2 kernel: [316382.943604] Total swap = 3444728kB Nov 29 17:51:01 virtuala2 kernel: [316382.950218] 524283 pages RAM Nov 29 17:51:01 virtuala2 kernel: [316382.950220] 9848 pages reserved Nov 29 17:51:01 virtuala2 kernel: [316382.950222] 400673 pages shared Nov 29 17:51:01 virtuala2 kernel: [316382.950223] 135939 pages non-shared Unfortunately this is a production machine so I cannot experiment with different kernels. If there's any other information I could provide that would help in correcting this bug, please let me know. -- crash after kswapd page allocation failure https://bugs.launchpad.net/bugs/661212 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
