On Tue, Jan 22, 2013 at 10:59:17AM +0100, Josip Rodin wrote: > I appear to be experiencing a serious problem with a 768 MB RAM Xen domU > machine running an NFS client - every now and then (for months now), often > in the middle of the night, it enters some kind of a broken state where a > few semi-random processes (mainly apache2's and vsftpd's which are told to > serve files from the NFS mount) start battling it out for the memory, and > everything including sshd starts invoking the OOM killer, over and over > again. Nothing seems to halt the downward spiral; manual invocation of the > OOM killer does nothing of any use. Terminating all processes is the only > thing that makes it go quiet, but then that's effectively the same as a > reboot. > > This is the SysRq+M output on the machine once it's been in the broken state > for a while: [...] > active_anon:394 inactive_anon:3197 isolated_anon:0 > active_file:25 inactive_file:176 isolated_file:32 > unevictable:2659 dirty:1 writeback:0 unstable:0 > free:21456 slab_reclaimable:16177 slab_unreclaimable:143165 > mapped:677 shmem:76 pagetables:455 bounce:0 [...] > The thing I noticed was the slab_unreclaimable explosion, by a factor > of 122. That... doesn't sound like something that should be happenning. > > Googling for slab_unreclaimable found me this old bug report about > slab_unreclaimable domU problems that was mass-closed with the switch to the > new paravirtops Xen release. Granted, our use case is not Samba like with > the original reporter, but the pattern of a file server was close enough for > me to be uncomfortable with it :|
I caught it earlier just now, at: [950084.590733] active_anon:2805 inactive_anon:11835 isolated_anon:0 [950084.590735] active_file:76 inactive_file:516 isolated_file:32 [950084.590737] unevictable:783 dirty:1 writeback:0 unstable:0 [950084.590739] free:26251 slab_reclaimable:15733 slab_unreclaimable:128868 [950084.590741] mapped:938 shmem:75 pagetables:651 bounce:0 And snuck in a few slabtops (even some -o invocations were getting killed, along with my shell and pretty much everything else): Active / Total Objects (% used) : 555753 / 587128 (94.7%) Active / Total Slabs (% used) : 49430 / 49430 (100.0%) Active / Total Caches (% used) : 65 / 76 (85.5%) Active / Total Size (% used) : 546613.78K / 553025.01K (98.8%) Minimum / Average / Maximum Object : 0.01K / 0.94K / 8.00K OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME 90993 66836 73% 0.19K 4333 21 17332K dentry 75840 73664 97% 0.12K 2370 32 9480K kmalloc-128 68096 68092 99% 0.01K 133 512 532K kmalloc-8 65888 65655 99% 0.25K 4118 16 16472K kmalloc-256 65820 65778 99% 1.00K 4767 16 76272K kmalloc-1024 65436 65414 99% 0.63K 5454 12 43632K proc_inode_cache 65419 65419 100% 4.00K 14179 8 453728K kmalloc-4096 65390 65390 100% 2.06K 13338 15 426816K net_namespace 4998 4990 99% 0.08K 98 51 392K sysfs_dir_cache 4224 2018 47% 0.06K 66 64 264K kmalloc-64 2288 2107 92% 0.18K 104 22 416K vm_area_struct 1792 1789 99% 0.02K 7 256 28K kmalloc-16 1470 1203 81% 0.19K 70 21 280K kmalloc-192 1300 402 30% 0.79K 65 20 1040K ext3_inode_cache 896 731 81% 0.03K 7 128 28K anon_vma 784 532 67% 0.55K 56 14 448K radix_tree_node A bit later: Active / Total Objects (% used) : 555403 / 586704 (94.7%) Active / Total Slabs (% used) : 49394 / 49394 (100.0%) Active / Total Caches (% used) : 65 / 76 (85.5%) Active / Total Size (% used) : 546552.82K / 552827.43K (98.9%) Minimum / Average / Maximum Object : 0.01K / 0.94K / 8.00K OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME 90993 66779 73% 0.19K 4333 21 17332K dentry 75840 73654 97% 0.12K 2370 32 9480K kmalloc-128 68096 68092 99% 0.01K 133 512 532K kmalloc-8 65888 65601 99% 0.25K 4118 16 16472K kmalloc-256 65852 65741 99% 1.00K 4760 16 76160K kmalloc-1024 65436 65409 99% 0.63K 5454 12 43632K proc_inode_cache 65428 65428 100% 4.00K 14181 8 453792K kmalloc-4096 65391 65391 100% 2.06K 13339 15 426848K net_namespace 4998 4986 99% 0.08K 98 51 392K sysfs_dir_cache 4224 2017 47% 0.06K 66 64 264K kmalloc-64 2134 2108 98% 0.18K 97 22 388K vm_area_struct 1792 1789 99% 0.02K 7 256 28K kmalloc-16 1449 1078 74% 0.19K 69 21 276K kmalloc-192 1100 376 34% 0.79K 55 20 880K ext3_inode_cache 896 639 71% 0.03K 7 128 28K anon_vma 714 554 77% 0.55K 51 14 408K radix_tree_node And a bit later still: [950506.398951] active_anon:2370 inactive_anon:12180 isolated_anon:2 [950506.398953] active_file:0 inactive_file:261 isolated_file:36 [950506.398955] unevictable:831 dirty:1 writeback:0 unstable:0 [950506.398957] free:26418 slab_reclaimable:15749 slab_unreclaimable:129161 [950506.398959] mapped:743 shmem:75 pagetables:636 bounce:0 Active / Total Objects (% used) : 556917 / 587606 (94.8%) Active / Total Slabs (% used) : 49717 / 49717 (100.0%) Active / Total Caches (% used) : 65 / 76 (85.5%) Active / Total Size (% used) : 548047.95K / 554096.57K (98.9%) Minimum / Average / Maximum Object : 0.01K / 0.94K / 8.00K OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME 90993 66921 73% 0.19K 4333 21 17332K dentry 75840 73825 97% 0.12K 2370 32 9480K kmalloc-128 68096 68092 99% 0.01K 133 512 532K kmalloc-8 65984 65766 99% 0.25K 4124 16 16496K kmalloc-256 65964 65913 99% 1.00K 4776 16 76416K kmalloc-1024 65592 65566 99% 0.63K 5467 12 43736K proc_inode_cache 65584 65584 100% 4.00K 14288 8 457216K kmalloc-4096 65552 65552 100% 2.06K 13500 15 432000K net_namespace 4998 4986 99% 0.08K 98 51 392K sysfs_dir_cache 4224 1976 46% 0.06K 66 64 264K kmalloc-64 2398 2398 100% 0.18K 109 22 436K vm_area_struct 1792 1789 99% 0.02K 7 256 28K kmalloc-16 1428 1112 77% 0.19K 68 21 272K kmalloc-192 1100 402 36% 0.79K 55 20 880K ext3_inode_cache 896 648 72% 0.03K 7 128 28K anon_vma 742 664 89% 0.55K 53 14 424K radix_tree_node [950532.846913] SysRq : Show Blocked State [950532.846930] task PC stack pid father [950532.846942] kswapd0 D ffff88002f123880 0 30 2 0x00000000 [950532.846955] ffff88002f123880 0000000000000246 ffff88001ae8cbd0 000000010e28d9ca [950532.846971] ffffffff8100e635 ffffffff8100ecf2 000000000000f9e0 ffff88002dfb1fd8 [950532.846987] 0000000000015780 0000000000015780 ffff88002fe44db0 ffff88002fe450a8 [950532.847003] Call Trace: [950532.847018] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa [950532.847028] [<ffffffff8100ecf2>] ? check_events+0x12/0x20 [950532.847037] [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1 [950532.847049] [<ffffffff8130e0ca>] ? _spin_unlock_irqrestore+0xd/0xe [950532.847061] [<ffffffff8105bf06>] ? __mod_timer+0x141/0x153 [950532.847071] [<ffffffff8130e0ca>] ? _spin_unlock_irqrestore+0xd/0xe [950532.847081] [<ffffffff8102ddc4>] ? pvclock_clocksource_read+0x3a/0x8b [950532.847091] [<ffffffff8130d164>] ? schedule_timeout+0xa5/0xdd [950532.847100] [<ffffffff8105ba80>] ? process_timeout+0x0/0x5 [950532.847110] [<ffffffff8130c4b7>] ? io_schedule_timeout+0x7e/0xca [950532.847121] [<ffffffff810c97bf>] ? congestion_wait+0x66/0x80 [950532.847131] [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e [950532.847142] [<ffffffff810c11a8>] ? kswapd+0x5f9/0x686 [950532.847151] [<ffffffff810be6ff>] ? isolate_pages_global+0x0/0x20f [950532.847160] [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e [950532.847170] [<ffffffff8100ecf2>] ? check_events+0x12/0x20 [950532.847179] [<ffffffff810c0baf>] ? kswapd+0x0/0x686 [950532.847187] [<ffffffff81065dc5>] ? kthread+0x79/0x81 [950532.847197] [<ffffffff81012baa>] ? child_rip+0xa/0x20 [950532.847205] [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b [950532.847214] [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6 [950532.847223] [<ffffffff81012ba0>] ? child_rip+0x0/0x20 [950532.847234] vsftpd D 0000000000000000 0 1325 1 0x00000000 [950532.847245] ffffffff814a11f0 0000000000000282 0000000000000000 000000010e28d9ca [950532.847261] ffffffff8100e635 ffffffff8100ecf2 000000000000f9e0 ffff880002697fd8 [950532.847276] 0000000000015780 0000000000015780 ffff88002f123880 ffff88002f123b78 [950532.847291] Call Trace: [950532.847299] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa [950532.847308] [<ffffffff8100ecf2>] ? check_events+0x12/0x20 [950532.847317] [<ffffffff8130e0ca>] ? _spin_unlock_irqrestore+0xd/0xe [950532.847327] [<ffffffff8105bf06>] ? __mod_timer+0x141/0x153 [950532.847336] [<ffffffff8130d164>] ? schedule_timeout+0xa5/0xdd [950532.848079] [<ffffffff8105ba80>] ? process_timeout+0x0/0x5 [950532.848088] [<ffffffff8130c4b7>] ? io_schedule_timeout+0x7e/0xca [950532.848098] [<ffffffff810c97bf>] ? congestion_wait+0x66/0x80 [950532.848108] [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e [950532.848117] [<ffffffff810c0105>] ? shrink_list+0x4a9/0x767 [950532.848127] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa [950532.848137] [<ffffffff8100ecf2>] ? check_events+0x12/0x20 [950532.848146] [<ffffffff810c0643>] ? shrink_zone+0x280/0x342 [950532.848156] [<ffffffff810c0846>] ? shrink_slab+0x141/0x153 [950532.848165] [<ffffffff810c170a>] ? try_to_free_pages+0x232/0x38e [950532.848175] [<ffffffff810be6ff>] ? isolate_pages_global+0x0/0x20f [950532.848187] [<ffffffff81154851>] ? cap_sb_alloc_security+0x0/0x3 [950532.848197] [<ffffffff810bb793>] ? __alloc_pages_nodemask+0x3d4/0x5fd [950532.848207] [<ffffffff810e8769>] ? kmem_cache_alloc+0x8c/0xf0 [950532.848216] [<ffffffff810e7f09>] ? new_slab+0x42/0x1ca [950532.848224] [<ffffffff810e8281>] ? __slab_alloc+0x1f0/0x39b [950532.848235] [<ffffffff8108bc1e>] ? copy_pid_ns+0x5f/0x2a5 [950532.848244] [<ffffffff8108bc1e>] ? copy_pid_ns+0x5f/0x2a5 [950532.848253] [<ffffffff810e875c>] ? kmem_cache_alloc+0x7f/0xf0 [950532.848262] [<ffffffff8108bc1e>] ? copy_pid_ns+0x5f/0x2a5 [950532.848271] [<ffffffff810692fa>] ? create_new_namespaces+0xc2/0x15e [950532.848281] [<ffffffff8106954e>] ? copy_namespaces+0x61/0x97 [950532.848291] [<ffffffff8104df25>] ? copy_process+0xa7a/0x1165 [950532.848300] [<ffffffff8104e767>] ? do_fork+0x157/0x31e [950532.848309] [<ffffffff813105a6>] ? do_page_fault+0x2e0/0x2fc [950532.848318] [<ffffffff81011e63>] ? stub_clone+0x13/0x20 [950532.848327] [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b [950532.848336] vsftpd D 0000000000000000 0 28790 28778 0x00000000 [950532.848348] ffffffff814a11f0 0000000000000282 0000000000000000 000000010e28d9d4 [950532.848363] ffffffff8100e635 ffffffff8100ecf2 000000000000f9e0 ffff8800279affd8 [950532.848379] 0000000000015780 0000000000015780 ffff88002f15c6a0 ffff88002f15c998 [950532.848395] Call Trace: [950532.848402] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa [950532.848412] [<ffffffff8100ecf2>] ? check_events+0x12/0x20 [950532.848421] [<ffffffff8130e0ca>] ? _spin_unlock_irqrestore+0xd/0xe [950532.848431] [<ffffffff8105bf06>] ? __mod_timer+0x141/0x153 [950532.848441] [<ffffffff8130d164>] ? schedule_timeout+0xa5/0xdd [950532.848450] [<ffffffff8105ba80>] ? process_timeout+0x0/0x5 [950532.848460] [<ffffffff8130c4b7>] ? io_schedule_timeout+0x7e/0xca [950532.848470] [<ffffffff810c97bf>] ? congestion_wait+0x66/0x80 [950532.848479] [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e [950532.848488] [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1 [950532.848498] [<ffffffff810bfd54>] ? shrink_list+0xf8/0x767 [950532.848509] [<ffffffff8119934b>] ? __bitmap_weight+0x3a/0x7e [950532.848519] [<ffffffff8118f7fe>] ? cpumask_next_and+0x2a/0x3a [950532.848529] [<ffffffff81045e7e>] ? find_busiest_group+0x420/0xa4e [950532.848539] [<ffffffff810bc2e5>] ? determine_dirtyable_memory+0xd/0x1d [950532.848548] [<ffffffff810bc35d>] ? get_dirty_limits+0x1d/0x259 [950532.848557] [<ffffffff81012cdb>] ? xen_hypervisor_callback+0x1b/0x20 [950532.848567] [<ffffffff8130e67a>] ? error_exit+0x2a/0x60 [950532.848575] [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6 [950532.848585] [<ffffffff810bc5c9>] ? throttle_vm_writeout+0x30/0x8d [950532.848595] [<ffffffff8100922a>] ? hypercall_page+0x22a/0x1001 [950532.848604] [<ffffffff810c0643>] ? shrink_zone+0x280/0x342 [950532.848614] [<ffffffff810c0846>] ? shrink_slab+0x141/0x153 [950532.848623] [<ffffffff810c170a>] ? try_to_free_pages+0x232/0x38e [950532.848633] [<ffffffff810be6ff>] ? isolate_pages_global+0x0/0x20f [950532.848643] [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1 [950532.848653] [<ffffffff810bb793>] ? __alloc_pages_nodemask+0x3d4/0x5fd [950532.848662] [<ffffffff810e7f09>] ? new_slab+0x42/0x1ca [950532.848671] [<ffffffff810e8281>] ? __slab_alloc+0x1f0/0x39b [950532.848681] [<ffffffff810fa746>] ? getname+0x23/0x1a0 [950532.848690] [<ffffffff810fa746>] ? getname+0x23/0x1a0 [950532.848699] [<ffffffff810e875c>] ? kmem_cache_alloc+0x7f/0xf0 [950532.848708] [<ffffffff810fa746>] ? getname+0x23/0x1a0 [950532.848717] [<ffffffff810fb1a6>] ? user_path_at+0x20/0x79 [950532.848726] [<ffffffff813105a6>] ? do_page_fault+0x2e0/0x2fc [950532.848735] [<ffffffff810ef186>] ? sys_chdir+0x1b/0x64 [950532.848744] [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b [950532.848753] apache2 D 0000000000000000 0 30698 18611 0x00000000 [950532.848765] ffff88002fdd0e20 0000000000000282 0000000000000000 000000010e28d9c8 [950532.848781] ffffffff8100e635 ffffffff8100ecf2 000000000000f9e0 ffff88000222ffd8 [950532.848796] 0000000000015780 0000000000015780 ffff880002699530 ffff880002699828 [950532.848812] Call Trace: [950532.848819] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa [950532.848829] [<ffffffff8100ecf2>] ? check_events+0x12/0x20 [950532.848838] [<ffffffff8130e0ca>] ? _spin_unlock_irqrestore+0xd/0xe [950532.848848] [<ffffffff8105bf06>] ? __mod_timer+0x141/0x153 [950532.848857] [<ffffffff8130d164>] ? schedule_timeout+0xa5/0xdd [950532.848867] [<ffffffff8105ba80>] ? process_timeout+0x0/0x5 [950532.848876] [<ffffffff8130c4b7>] ? io_schedule_timeout+0x7e/0xca [950532.848886] [<ffffffff810c97bf>] ? congestion_wait+0x66/0x80 [950532.848895] [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e [950532.848905] [<ffffffff810c0105>] ? shrink_list+0x4a9/0x767 [950532.848915] [<ffffffff81054915>] ? _local_bh_enable_ip+0x22/0x8f [950532.848925] [<ffffffff81054915>] ? _local_bh_enable_ip+0x22/0x8f [950532.848935] [<ffffffff810bc2e5>] ? determine_dirtyable_memory+0xd/0x1d [950532.848944] [<ffffffff810bc35d>] ? get_dirty_limits+0x1d/0x259 [950532.848954] [<ffffffff81054915>] ? _local_bh_enable_ip+0x22/0x8f [950532.848964] [<ffffffff810c0643>] ? shrink_zone+0x280/0x342 [950532.848975] [<ffffffff810c0846>] ? shrink_slab+0x141/0x153 [950532.848985] [<ffffffff810c170a>] ? try_to_free_pages+0x232/0x38e [950532.848995] [<ffffffff810be6ff>] ? isolate_pages_global+0x0/0x20f [950532.849004] [<ffffffff810bb793>] ? __alloc_pages_nodemask+0x3d4/0x5fd [950532.849014] [<ffffffff8100ecf2>] ? check_events+0x12/0x20 [950532.849024] [<ffffffff8129835d>] ? tcp_established_options+0x2d/0xa9 [950532.849033] [<ffffffff810e7f09>] ? new_slab+0x42/0x1ca [950532.849041] [<ffffffff810e8281>] ? __slab_alloc+0x1f0/0x39b [950532.849051] [<ffffffff81290081>] ? sk_stream_alloc_skb+0x2f/0xd5 [950532.849061] [<ffffffff810e8cb3>] ? __kmalloc_node_track_caller+0xbb/0x11b [950532.849070] [<ffffffff81290081>] ? sk_stream_alloc_skb+0x2f/0xd5 [950532.849081] [<ffffffff8125932f>] ? __alloc_skb+0x69/0x15a [950532.849090] [<ffffffff8110b0bc>] ? pipe_to_sendpage+0x0/0x74 [950532.849099] [<ffffffff81290081>] ? sk_stream_alloc_skb+0x2f/0xd5 [950532.849109] [<ffffffff812902d3>] ? tcp_sendmsg+0x1ac/0x87f [950532.849118] [<ffffffff81041432>] ? set_next_entity+0x34/0x56 [950532.849127] [<ffffffff8110b0bc>] ? pipe_to_sendpage+0x0/0x74 [950532.849138] [<ffffffff81251c65>] ? sock_sendmsg+0xa3/0xbb [950532.849146] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa [950532.849156] [<ffffffff8100ecf2>] ? check_events+0x12/0x20 [950532.849165] [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1 [950532.849174] [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e [950532.849183] [<ffffffff8130ce2a>] ? io_schedule+0x93/0xb7 [950532.849192] [<ffffffff8130d251>] ? __wait_on_bit_lock+0x76/0x84 [950532.849202] [<ffffffff8110d00c>] ? __generic_file_splice_read+0x36e/0x3bd [950532.849211] [<ffffffff81251f64>] ? kernel_sendmsg+0x32/0x3f [950532.849220] [<ffffffff81254790>] ? sock_no_sendpage+0x87/0x94 [950532.849230] [<ffffffff812909f1>] ? tcp_sendpage+0x4b/0x458 [950532.849239] [<ffffffff8130d251>] ? __wait_on_bit_lock+0x76/0x84 [950532.849248] [<ffffffff8110b0bc>] ? pipe_to_sendpage+0x0/0x74 [950532.850903] [<ffffffff81250d0d>] ? kernel_sendpage+0x16/0x1f [950532.850903] [<ffffffff81250d4b>] ? sock_sendpage+0x35/0x3a [950532.850903] [<ffffffff8110b122>] ? pipe_to_sendpage+0x66/0x74 [950532.850903] [<ffffffff8110b17c>] ? splice_from_pipe_feed+0x4c/0xd4 [950532.850903] [<ffffffff8110b0bc>] ? pipe_to_sendpage+0x0/0x74 [950532.850903] [<ffffffff8110b405>] ? __splice_from_pipe+0x3e/0x69 [950532.850903] [<ffffffff8110b0bc>] ? pipe_to_sendpage+0x0/0x74 [950532.850903] [<ffffffff8110b47d>] ? splice_from_pipe+0x4d/0x63 [950532.850903] [<ffffffff8110c00d>] ? direct_splice_actor+0x1b/0x1e [950532.850903] [<ffffffff8110b9c8>] ? splice_direct_to_actor+0xdc/0x188 [950532.850903] [<ffffffff8110bff2>] ? direct_splice_actor+0x0/0x1e [950532.850903] [<ffffffff8110babe>] ? do_splice_direct+0x4a/0x64 [950532.850903] [<ffffffff810f0312>] ? do_sendfile+0x12d/0x1a8 [950532.850903] [<ffffffff810f03d6>] ? sys_sendfile64+0x49/0x89 [950532.850903] [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b [950532.850903] apache2 D 0000000000000000 0 30702 18611 0x00000000 [950532.850903] ffffffff814a11f0 0000000000000282 0000000000000000 000000010e28d9c8 [950532.850903] ffffffff8100e635 ffffffff8100ecf2 000000000000f9e0 ffff880023c6bfd8 [950532.850903] 0000000000015780 0000000000015780 ffff880002503880 ffff880002503b78 [950532.850903] Call Trace: [950532.850903] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa [950532.850903] [<ffffffff8100ecf2>] ? check_events+0x12/0x20 [950532.850903] [<ffffffff8130e0ca>] ? _spin_unlock_irqrestore+0xd/0xe [950532.850903] [<ffffffff8105bf06>] ? __mod_timer+0x141/0x153 [950532.850903] [<ffffffff8130d164>] ? schedule_timeout+0xa5/0xdd [950532.850903] [<ffffffff8105ba80>] ? process_timeout+0x0/0x5 [950532.850903] [<ffffffff8130c4b7>] ? io_schedule_timeout+0x7e/0xca [950532.850903] [<ffffffff810c97bf>] ? congestion_wait+0x66/0x80 [950532.850903] [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e [950532.850903] [<ffffffff810c0105>] ? shrink_list+0x4a9/0x767 [950532.850903] [<ffffffff8130e0ca>] ? _spin_unlock_irqrestore+0xd/0xe [950532.850903] [<ffffffff81054915>] ? _local_bh_enable_ip+0x22/0x8f [950532.850903] [<ffffffff810bc2e5>] ? determine_dirtyable_memory+0xd/0x1d [950532.850903] [<ffffffff810bc35d>] ? get_dirty_limits+0x1d/0x259 [950532.850903] [<ffffffff81054915>] ? _local_bh_enable_ip+0x22/0x8f [950532.850903] [<ffffffff810c0643>] ? shrink_zone+0x280/0x342 [950532.850903] [<ffffffff810c0846>] ? shrink_slab+0x141/0x153 [950532.850903] [<ffffffff810c170a>] ? try_to_free_pages+0x232/0x38e [950532.850903] [<ffffffff810be6ff>] ? isolate_pages_global+0x0/0x20f [950532.850903] [<ffffffff810bb793>] ? __alloc_pages_nodemask+0x3d4/0x5fd [950532.850903] [<ffffffff8100922a>] ? hypercall_page+0x22a/0x1001 [950532.850903] [<ffffffff810e7f09>] ? new_slab+0x42/0x1ca [950532.850903] [<ffffffff810e8281>] ? __slab_alloc+0x1f0/0x39b [950532.850903] [<ffffffff81290081>] ? sk_stream_alloc_skb+0x2f/0xd5 [950532.850903] [<ffffffff810e8cb3>] ? __kmalloc_node_track_caller+0xbb/0x11b [950532.850903] [<ffffffff81290081>] ? sk_stream_alloc_skb+0x2f/0xd5 [950532.850903] [<ffffffff8125932f>] ? __alloc_skb+0x69/0x15a [950532.850903] [<ffffffff8110b0bc>] ? pipe_to_sendpage+0x0/0x74 [950532.850903] [<ffffffff81290081>] ? sk_stream_alloc_skb+0x2f/0xd5 [950532.850903] [<ffffffff812902d3>] ? tcp_sendmsg+0x1ac/0x87f [950532.850903] [<ffffffff8100ecf2>] ? check_events+0x12/0x20 [950532.850903] [<ffffffff8110b0bc>] ? pipe_to_sendpage+0x0/0x74 [950532.850903] [<ffffffff81251c65>] ? sock_sendmsg+0xa3/0xbb [950532.850903] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa [950532.850903] [<ffffffff8100ecf2>] ? check_events+0x12/0x20 [950532.850903] [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1 [950532.850903] [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e [950532.850903] [<ffffffff8130ce2a>] ? io_schedule+0x93/0xb7 [950532.850903] [<ffffffff8130d251>] ? __wait_on_bit_lock+0x76/0x84 [950532.850903] [<ffffffff8110d00c>] ? __generic_file_splice_read+0x36e/0x3bd [950532.850903] [<ffffffff81251f64>] ? kernel_sendmsg+0x32/0x3f [950532.850903] [<ffffffff81254790>] ? sock_no_sendpage+0x87/0x94 [950532.850903] [<ffffffff812909f1>] ? tcp_sendpage+0x4b/0x458 [950532.850903] [<ffffffff81012cdb>] ? xen_hypervisor_callback+0x1b/0x20 [950532.850903] [<ffffffff8110b0bc>] ? pipe_to_sendpage+0x0/0x74 [950532.850903] [<ffffffff81250d0d>] ? kernel_sendpage+0x16/0x1f [950532.850903] [<ffffffff81250d4b>] ? sock_sendpage+0x35/0x3a [950532.850903] [<ffffffff8110b122>] ? pipe_to_sendpage+0x66/0x74 [950532.850903] [<ffffffff8110b17c>] ? splice_from_pipe_feed+0x4c/0xd4 [950532.850903] [<ffffffff8110b0bc>] ? pipe_to_sendpage+0x0/0x74 [950532.850903] [<ffffffff8110b405>] ? __splice_from_pipe+0x3e/0x69 [950532.850903] [<ffffffff8110b0bc>] ? pipe_to_sendpage+0x0/0x74 [950532.850903] [<ffffffff8110b47d>] ? splice_from_pipe+0x4d/0x63 [950532.850903] [<ffffffff8110c00d>] ? direct_splice_actor+0x1b/0x1e [950532.850903] [<ffffffff8110b9c8>] ? splice_direct_to_actor+0xdc/0x188 [950532.850903] [<ffffffff8110bff2>] ? direct_splice_actor+0x0/0x1e [950532.850903] [<ffffffff8110babe>] ? do_splice_direct+0x4a/0x64 [950532.850903] [<ffffffff810f0312>] ? do_sendfile+0x12d/0x1a8 [950532.850903] [<ffffffff810f03d6>] ? sys_sendfile64+0x49/0x89 [950532.850903] [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b [950532.850903] vsftpd D 0000000000000000 0 32054 32049 0x00000000 [950532.850903] ffffffff814a11f0 0000000000000286 0000000000000000 000000010e28d9dd [950532.850903] ffffffff8100e635 ffffffff8100ecf2 000000000000f9e0 ffff8800241e3fd8 [950532.850903] 0000000000015780 0000000000015780 ffff880002698710 ffff880002698a08 [950532.850903] Call Trace: [950532.850903] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa [950532.850903] [<ffffffff8100ecf2>] ? check_events+0x12/0x20 [950532.850903] [<ffffffff8130e0ca>] ? _spin_unlock_irqrestore+0xd/0xe [950532.850903] [<ffffffff8105bf06>] ? __mod_timer+0x141/0x153 [950532.850903] [<ffffffff8130d164>] ? schedule_timeout+0xa5/0xdd [950532.850903] [<ffffffff8105ba80>] ? process_timeout+0x0/0x5 [950532.850903] [<ffffffff8130c4b7>] ? io_schedule_timeout+0x7e/0xca [950532.850903] [<ffffffff810c97bf>] ? congestion_wait+0x66/0x80 [950532.850903] [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e [950532.850903] [<ffffffff810c0105>] ? shrink_list+0x4a9/0x767 [950532.850903] [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1 [950532.850903] [<ffffffff8130e0ca>] ? _spin_unlock_irqrestore+0xd/0xe [950532.850903] [<ffffffffa00bb1ef>] ? tcp_packet+0xb20/0xb65 [nf_conntrack] [950532.850903] [<ffffffff810bc2e5>] ? determine_dirtyable_memory+0xd/0x1d [950532.850903] [<ffffffff810bc35d>] ? get_dirty_limits+0x1d/0x259 [950532.850903] [<ffffffff81054915>] ? _local_bh_enable_ip+0x22/0x8f [950532.850903] [<ffffffffa01aface>] ? ipt_do_table+0x5ee/0x620 [ip_tables] [950532.850903] [<ffffffff810c0643>] ? shrink_zone+0x280/0x342 [950532.850903] [<ffffffff810c0846>] ? shrink_slab+0x141/0x153 [950532.850903] [<ffffffff810c170a>] ? try_to_free_pages+0x232/0x38e [950532.850903] [<ffffffff810be6ff>] ? isolate_pages_global+0x0/0x20f [950532.850903] [<ffffffff810bb793>] ? __alloc_pages_nodemask+0x3d4/0x5fd [950532.850903] [<ffffffff810e7f09>] ? new_slab+0x42/0x1ca [950532.850903] [<ffffffff810e8281>] ? __slab_alloc+0x1f0/0x39b [950532.850903] [<ffffffff81290081>] ? sk_stream_alloc_skb+0x2f/0xd5 [950532.850903] [<ffffffff810e8cb3>] ? __kmalloc_node_track_caller+0xbb/0x11b [950532.850903] [<ffffffff81290081>] ? sk_stream_alloc_skb+0x2f/0xd5 [950532.850903] [<ffffffff8125932f>] ? __alloc_skb+0x69/0x15a [950532.850903] [<ffffffff81290081>] ? sk_stream_alloc_skb+0x2f/0xd5 [950532.850903] [<ffffffff812902d3>] ? tcp_sendmsg+0x1ac/0x87f [950532.850903] [<ffffffff812514dc>] ? sock_aio_write+0xb1/0xbc [950532.850903] [<ffffffff8100c426>] ? pfn_to_mfn+0xe/0x22 [950532.850903] [<ffffffff8100cc43>] ? xen_make_pte+0x7b/0x83 [950532.850903] [<ffffffff810cbeee>] ? __do_fault+0x38c/0x3c3 [950532.850903] [<ffffffff810effee>] ? do_sync_write+0xce/0x113 [950532.850903] [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e [950532.850903] [<ffffffff810f0953>] ? vfs_write+0xbc/0x102 [950532.850903] [<ffffffff810f0a55>] ? sys_write+0x45/0x6e [950532.850903] [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b [950532.850903] update-pkg-ft D 0000000000000000 0 32082 32038 0x00000000 [950532.850903] ffffffff814a11f0 0000000000000282 0000000000000000 000000010e28d9d2 [950532.850903] ffffffff8100e635 ffffffff8100ecf2 000000000000f9e0 ffff880024f07fd8 [950532.850903] 0000000000015780 0000000000015780 ffff880002798000 ffff8800027982f8 [950532.850903] Call Trace: [950532.850903] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa [950532.850903] [<ffffffff8100ecf2>] ? check_events+0x12/0x20 [950532.850903] [<ffffffff8130e0ca>] ? _spin_unlock_irqrestore+0xd/0xe [950532.850903] [<ffffffff8105bf06>] ? __mod_timer+0x141/0x153 [950532.850903] [<ffffffff8130d164>] ? schedule_timeout+0xa5/0xdd [950532.850903] [<ffffffff8105ba80>] ? process_timeout+0x0/0x5 [950532.850903] [<ffffffffa0038d2f>] ? bdev_try_to_free_page+0x0/0x64 [ext3] [950532.850903] [<ffffffff8130c4b7>] ? io_schedule_timeout+0x7e/0xca [950532.850903] [<ffffffff810c97bf>] ? congestion_wait+0x66/0x80 [950532.850903] [<ffffffff81066092>] ? autoremove_wake_function+0x0/0x2e [950532.850903] [<ffffffff810c0105>] ? shrink_list+0x4a9/0x767 [950532.850903] [<ffffffff810bc2e5>] ? determine_dirtyable_memory+0xd/0x1d [950532.850903] [<ffffffff810bc35d>] ? get_dirty_limits+0x1d/0x259 [950532.850903] [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1 [950532.850903] [<ffffffff810993bc>] ? __call_rcu+0x110/0x118 [950532.850903] [<ffffffff81012cdb>] ? xen_hypervisor_callback+0x1b/0x20 [950532.850903] [<ffffffff810c0643>] ? shrink_zone+0x280/0x342 [950532.850903] [<ffffffff810c0846>] ? shrink_slab+0x141/0x153 [950532.850903] [<ffffffff810c170a>] ? try_to_free_pages+0x232/0x38e [950532.850903] [<ffffffff810be6ff>] ? isolate_pages_global+0x0/0x20f [950532.850903] [<ffffffff810bb793>] ? __alloc_pages_nodemask+0x3d4/0x5fd [950532.850903] [<ffffffff810e7f09>] ? new_slab+0x42/0x1ca [950532.850903] [<ffffffff810e8281>] ? __slab_alloc+0x1f0/0x39b [950532.850903] [<ffffffff810fa746>] ? getname+0x23/0x1a0 [950532.850903] [<ffffffff810fa746>] ? getname+0x23/0x1a0 [950532.850903] [<ffffffff810e875c>] ? kmem_cache_alloc+0x7f/0xf0 [950532.850903] [<ffffffff810fa746>] ? getname+0x23/0x1a0 [950532.850903] [<ffffffff810fb1a6>] ? user_path_at+0x20/0x79 [950532.850903] [<ffffffff810cdf3c>] ? handle_mm_fault+0x3b8/0x80f [950532.850903] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa [950532.850903] [<ffffffff810f361a>] ? vfs_fstatat+0x2c/0x57 [950532.850903] [<ffffffff8100e635>] ? xen_force_evtchn_callback+0x9/0xa [950532.850903] [<ffffffff810f370d>] ? sys_newstat+0x11/0x30 [950532.850903] [<ffffffff8100ecdf>] ? xen_restore_fl_direct_end+0x0/0x1 [950532.850903] [<ffffffff8100b4d3>] ? xen_mc_issue+0x21/0x23 [950532.850903] [<ffffffff8101295b>] ? device_not_available+0x1b/0x20 [950532.850903] [<ffffffff81011b42>] ? system_call_fastpath+0x16/0x1b [950532.850903] Sched Debug Version: v0.09, 2.6.32-5-xen-amd64 #1 [950532.850903] now at 950533247.852699 msecs [950532.850903] .jiffies : 4532525608 [950532.850903] .sysctl_sched_latency : 10.000000 [950532.850903] .sysctl_sched_min_granularity : 2.000000 [950532.850903] .sysctl_sched_wakeup_granularity : 2.000000 [950532.850903] .sysctl_sched_child_runs_first : 0.000000 [950532.850903] .sysctl_sched_features : 32611451 [950532.850903] [950532.850903] cpu#0, 2333.472 MHz [950532.850903] .nr_running : 0 [950532.850903] .load : 0 [950532.850903] .nr_switches : 238671752 [950532.850903] .nr_load_updates : 165114440 [950532.850903] .nr_uninterruptible : 6 [950532.850903] .next_balance : 4532.525542 [950532.850903] .curr->pid : 0 [950532.850903] .clock : 950532844.214897 [950532.850903] .cpu_load[0] : 0 [950532.850903] .cpu_load[1] : 192 [950532.850903] .cpu_load[2] : 254 [950532.850903] .cpu_load[3] : 197 [950532.850903] .cpu_load[4] : 130 [950532.850903] [950532.850903] cfs_rq[0]:/ [950532.850903] .exec_clock : 0.000000 [950532.850903] .MIN_vruntime : 0.000001 [950532.850903] .min_vruntime : 136491473.007815 [950532.850903] .max_vruntime : 0.000001 [950532.850903] .spread : 0.000000 [950532.850903] .spread0 : 0.000000 [950532.850903] .nr_running : 0 [950532.850903] .load : 0 [950532.850903] .nr_spread_over : 0 [950532.850903] .shares : 0 [950532.850903] [950532.850903] rt_rq[0]: [950532.850903] .rt_nr_running : 0 [950532.850903] .rt_throttled : 0 [950532.850903] .rt_time : 0.000000 [950532.850903] .rt_runtime : 950.000000 [950532.850903] [950532.850903] runnable tasks: [950532.850903] task PID tree-key switches prio exec-runtime sum-exec sum-sleep [950532.850903] ---------------------------------------------------------------------------------------------------------- [950532.850903] [950532.850903] cpu#1, 2333.472 MHz [950532.850903] .nr_running : 3 [950532.850903] .load : 3072 [950532.850903] .nr_switches : 45263996 [950532.850903] .nr_load_updates : 121842283 [950532.850903] .nr_uninterruptible : 0 [950532.850903] .next_balance : 4532.525625 [950532.850903] .curr->pid : 32083 [950532.850903] .clock : 950533245.009907 [950532.850903] .cpu_load[0] : 3072 [950532.850903] .cpu_load[1] : 3072 [950532.850903] .cpu_load[2] : 3072 [950532.850903] .cpu_load[3] : 3072 [950532.850903] .cpu_load[4] : 3048 [950532.850903] [950532.850903] cfs_rq[1]:/ [950532.850903] .exec_clock : 0.000000 [950532.850903] .MIN_vruntime : 161709543.323397 [950532.850903] .min_vruntime : 161709548.323397 [950532.850903] .max_vruntime : 161709543.323397 [950532.850903] .spread : 0.000000 [950532.850903] .spread0 : 25218075.315582 [950532.850903] .nr_running : 3 [950532.850903] .load : 3072 [950532.850903] .nr_spread_over : 0 [950532.850903] .shares : 0 [950532.850903] [950532.850903] rt_rq[1]: [950532.850903] .rt_nr_running : 0 [950532.850903] .rt_throttled : 0 [950532.850903] .rt_time : 0.000000 [950532.850903] .rt_runtime : 950.000000 [950532.850903] [950532.850903] runnable tasks: [950532.850903] task PID tree-key switches prio exec-runtime sum-exec sum-sleep [950532.850903] ---------------------------------------------------------------------------------------------------------- [950532.850903] events/1 10 161709543.323397 1041646 120 0 0 0.000000 0.000000 0.000000 / [950532.850903] apache2 30698 161709543.323397 3096 120 0 0 0.000000 0.000000 0.000000 / [950532.850903] R network.sh 32083 161709832.320247 2 120 0 0 0.000000 0.000000 0.000000 / [950532.850903] -- 2. That which causes joy or happiness. -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org