Its not actually a bug in kernel but a bug in the application that is
trying to acquire large size memory pages and kernel is failing to
provide those pages. In our case the application that was misbehaving
was the device driver of our network card. Try updating the nic card
driver on your system and see if that helps.
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux-ec2 in Ubuntu.
https://bugs.launchpad.net/bugs/648721
Title:
page allocation failure on machines under heavy network load
Status in linux-ec2 package in Ubuntu:
Invalid
Bug description:
Tuning option:
The error messages seen below indicate failure to allocate a 32K chunk
of memory (order 5 allocation = 2^5K), the valid function calls in the
stack trace lead to the network stack. The problem mainly is that the
virtual network devices allocate and free buffers quicker than the vm
subsystem can keep up with. Increasing the amount of memory tried to
keep free for allocations (/proc/sys/vm/min_free_kbytes) avoids the
messages.
---
This error occurs in many processes running on our web servers:
apache2, glusterfsd, kswapd0, rsyslogd, swapper and zabbix_agentd
today.
Sep 27 07:39:19 web3 kernel: [2909159.608929] __ratelimit: 35 callbacks
suppressed
Sep 27 07:39:19 web3 kernel: [2909159.608933] swapper: page allocation
failure. order:5, mode:0x20
Sep 27 07:39:19 web3 kernel: [2909159.608937] Pid: 0, comm: swapper Not
tainted 2.6.32-305-ec2 #9-Ubuntu
Sep 27 07:39:19 web3 kernel: [2909159.608939] Call Trace:
Sep 27 07:39:19 web3 kernel: [2909159.608948] [] ? printk+0x18/0x1b
Sep 27 07:39:19 web3 kernel: [2909159.608960] []
__alloc_pages_slowpath+0x46e/0x4a0
Sep 27 07:39:19 web3 kernel: [2909159.608963] []
__alloc_pages_nodemask+0x14d/0x160
Sep 27 07:39:19 web3 kernel: [2909159.608968] [] T.685+0xe5/0x3a0
Sep 27 07:39:19 web3 kernel: [2909159.608971] []
cache_alloc_refill+0x1fc/0x230
Sep 27 07:39:19 web3 kernel: [2909159.608975] []
__kmalloc+0x176/0x1a0
Sep 27 07:39:19 web3 kernel: [2909159.608979] []
pskb_expand_head+0x53/0x170
Sep 27 07:39:19 web3 kernel: [2909159.608982] []
__pskb_pull_tail+0x4c/0x2d0
Sep 27 07:39:19 web3 kernel: [2909159.608987] [] ?
nf_iterate+0x71/0x90
Sep 27 07:39:19 web3 kernel: [2909159.608990] [] ?
dst_output+0x0/0x10
Sep 27 07:39:19 web3 kernel: [2909159.608993] []
dev_queue_xmit+0x143/0x560
Sep 27 07:39:19 web3 kernel: [2909159.608996] [] ?
nf_hook_slow+0x59/0xd0
Sep 27 07:39:19 web3 kernel: [2909159.609000] []
ip_finish_output+0x12f/0x2c0
Sep 27 07:39:19 web3 kernel: [2909159.609003] []
ip_output+0x9f/0xb0
Sep 27 07:39:19 web3 kernel: [2909159.609006] [] ?
dst_output+0x0/0x10
Sep 27 07:39:19 web3 kernel: [2909159.609008] []
ip_local_out+0x18/0x20
Sep 27 07:39:19 web3 kernel: [2909159.609011] []
ip_queue_xmit+0x147/0x390
Sep 27 07:39:19 web3 kernel: [2909159.609016] [] ?
enqueue_entity+0x14c/0x1e0
Sep 27 07:39:19 web3 kernel: [2909159.609022] [] ?
lock_timer_base+0x27/0x50
Sep 27 07:39:19 web3 kernel: [2909159.609027] []
tcp_transmit_skb+0x395/0x6e0
Sep 27 07:39:19 web3 kernel: [2909159.609030] [] ?
tcp_clean_rtx_queue+0x5af/0x790
Sep 27 07:39:19 web3 kernel: [2909159.609033] []
tcp_write_xmit+0x167/0x460
Sep 27 07:39:19 web3 kernel: [2909159.609037] []
__tcp_push_pending_frames+0x31/0x90
Sep 27 07:39:19 web3 kernel: [2909159.609040] []
tcp_data_snd_check+0x26/0xe0
Sep 27 07:39:19 web3 kernel: [2909159.609043] []
tcp_rcv_established+0x2eb/0x5d0
Sep 27 07:39:19 web3 kernel: [2909159.609046] []
tcp_v4_do_rcv+0xbd/0x130
Sep 27 07:39:19 web3 kernel: [2909159.609049] []
tcp_v4_rcv+0x5fe/0x6f0
Sep 27 07:39:19 web3 kernel: [2909159.609051] [] ?
nf_iterate+0x71/0x90
Sep 27 07:39:19 web3 kernel: [2909159.609054] [] ?
nf_hook_slow+0x59/0xd0
Sep 27 07:39:19 web3 kernel: [2909159.609058] []
ip_local_deliver_finish+0xa7/0x270
Sep 27 07:39:19 web3 kernel: [2909159.609061] []
ip_local_deliver+0x87/0x90
Sep 27 07:39:19 web3 kernel: [2909159.609064] [] ?
ip_local_deliver_finish+0x0/0x270
Sep 27 07:39:19 web3 kernel: [2909159.609067] []
ip_rcv_finish+0xeb/0x380
Sep 27 07:39:19 web3 kernel: [2909159.609071] [] ?
packet_rcv_spkt+0x38/0x140
Sep 27 07:39:19 web3 kernel: [2909159.609074] [] ip_rcv+0x21e/0x2e0
Sep 27 07:39:19 web3 kernel: [2909159.609077] [] ?
packet_rcv_spkt+0x38/0x140
Sep 27 07:39:19 web3 kernel: [2909159.609080] []
netif_receive_skb+0x39f/0x5c0
Sep 27 07:39:19 web3 kernel: [2909159.609084] []
netif_poll+0x4ce/0x7b0
Sep 27 07:39:19 web3 kernel: [2909159.609088] []
net_rx_action+0x12a/0x270
Sep 27 07:39:19 web3 kernel: [2909159.609091] [] ?
tcp_write_timer+0x0/0x200
Sep 27 07:39:19 web3 kernel: [2909159.609095] []
__do_softirq+0xab/0x1e0
Sep 27 07:39:19 web3 kernel: [2909159.609099] [] ?
handle_IRQ_event+0x97/0x180
Sep 27 07:39:19 web3 kernel: [2909159.6091
