Re: 2.6.23-rc8 network problem. Mem leak? ip1000a?
> OK. Did you try to reproduce it without the pps patch applied? No. But I've yanked the ip1000a driver (using old crufy vendor-supplied out-of-kernel module) and the problems are GONE. >> This machine receives more data than it sends, so I'd expect acks to >> outnumber "real" packets. Could the ip1000a driver's transmit path be >> leaking skbs somehow? > Absolutely. Normally a driver's transmit completion interrupt handler will > run dev_kfree_skb_irq() against the skbs which have been fully sent. > > However it'd be darned odd if the driver was leaking only tcp acks. It's leaking lots of things... you can see ARP packets in there and all sorts of stuff. But the big traffic hog is BackupPC doing inbound rsyncs all night long, which generates a lot of acks. Those are the packets it sends, so those are the packets that get leaked. > I can find no occurrence of "dev_kfree_skb" in drivers/net/ipg.c, which is > suspicious. Look for "IPG_DEV_KFREE_SKB", which is a wrapper macro. (Or just add "-i" to your grep.) It should probably be deleted (it just expands to dev_kfree_skb), but was presumably useful to someone during development. > Where did you get your ipg.c from, btw? davem's tree? rc8-mm1? rc8-mm2?? As I wrote originally, I got it from http://marc.info/?l=linux-netdev=118980588419882 which was a reuqest for mainline submission. If there are other patches floating around, I'm happy to try them. Now that I know what to look for, it's easy to spot the leak before OOM. > I assume that meminfo was not captured when the system was ooming? There > isn't much slab there. Oops, sorry. I captured slabinfo but not meminfo. Thank you very much! Sorry to jump the gun and post a lot before I had all the data, but if it WAS a problem in -rc8, I wanted to mention it before -final. Now, the rush is to get the ip1000a driver fixed before the merge window opens. I've added all the ip1000a developers to the Cc: list in an attempt to speed that up. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: 2.6.23-rc8 network problem. Mem leak? ip1000a?
On 30 Sep 2007 03:59:56 -0400 [EMAIL PROTECTED] wrote: > > ntpd. Sounds like pps leaking to me. > > That's what I'd think, except that pps does no allocation in the normal > running state, so there's nothing to leak. The interrupt path just > records the time in some preallocated, static buffers and wakes up > blocked readers. The read path copies the latest data out of those > static buffers. There's allocation when the PPS device is created, > and more when it's opened. OK. Did you try to reproduce it without the pps patch applied? > >> Can anyone offer some diagnosis advice? > > > CONFIG_DEBUG_SLAB_LEAK? > > Ah, thanks you; I've been using SLUB which doesn't support this option. > Here's what I've extracted. I've only presented the top few > slab_allocators and a small subset of the oom-killer messages, but I > have full copies if desired. Unfortunately, I've discovered that the > machine doesn't live in this unhappy state forever. Indeed, I'm not > sure if killing ntpd "fixes" anything; my previous observations > may have been optimistic ignorance. > > (For my own personal reference looking for more oom-kill, I nuked ntpd > at 06:46:56. And the oom-kills are continuing, with the latest at > 07:43:52.) > > Anyway, I have a bunch of information from the slab_allocators file, but > I'm not quire sure how to make sense of it. > > > With a machine in the unhappy state and firing the OOM killer, the top > 20 slab_allocators are: > $ sort -rnk2 /proc/slab_allocators | head -20 > skbuff_head_cache: 1712746 __alloc_skb+0x31/0x121 > size-512: 1706572 tcp_send_ack+0x23/0x102 > skbuff_fclone_cache: 149113 __alloc_skb+0x31/0x121 > size-2048: 148500 tcp_sendmsg+0x1b5/0xae1 > sysfs_dir_cache: 5289 sysfs_new_dirent+0x4b/0xec > size-512: 2613 sock_alloc_send_skb+0x93/0x1dd > Acpi-Operand: 2014 acpi_ut_allocate_object_desc_dbg+0x34/0x6e > size-32: 1995 sysfs_new_dirent+0x29/0xec > vm_area_struct: 1679 mmap_region+0x18f/0x421 > size-512: 1618 tcp_xmit_probe_skb+0x1f/0xcd > size-512: 1571 arp_create+0x4e/0x1cd > vm_area_struct: 1544 copy_process+0x9f1/0x1108 > anon_vma: 1448 anon_vma_prepare+0x29/0x74 > filp: 1201 get_empty_filp+0x44/0xcd > UDP: 1173 sk_alloc+0x25/0xaf > size-128: 1048 r1bio_pool_alloc+0x23/0x3b > size-128: 1024 nfsd_cache_init+0x2d/0xcf > Acpi-Namespace: 973 acpi_ns_create_node+0x2c/0x45 > vm_area_struct: 717 split_vma+0x33/0xe5 > dentry: 594 d_alloc+0x24/0x177 > > I'm not sure quite what "normal" numbers are, but I do wonder why there > are 1.7 million TCP acks buffered in the system. Shouldn't they be > transmitted and deallocated pretty quickly? Yeah, that's an skbuff leak. > This machine receives more data than it sends, so I'd expect acks to > outnumber "real" packets. Could the ip1000a driver's transmit path be > leaking skbs somehow? Absolutely. Normally a driver's transmit completion interrupt handler will run dev_kfree_skb_irq() against the skbs which have been fully sent. However it'd be darned odd if the driver was leaking only tcp acks. I can find no occurrence of "dev_kfree_skb" in drivers/net/ipg.c, which is suspicious. Where did you get your ipg.c from, btw? davem's tree? rc8-mm1? rc8-mm2?? > that would also explain the "flailing" of the > oom-killer; it can't associate the allocations with a process. > > Here's /proc/meminfo: > MemTotal: 1035756 kB > MemFree: 43508 kB > Buffers: 72920 kB > Cached: 224056 kB > SwapCached: 344916 kB > Active: 664976 kB > Inactive: 267656 kB > SwapTotal: 4950368 kB > SwapFree: 3729384 kB > Dirty:6460 kB > Writeback: 0 kB > AnonPages: 491708 kB > Mapped: 79232 kB > Slab:41324 kB > SReclaimable:25008 kB > SUnreclaim: 16316 kB > PageTables: 8132 kB > NFS_Unstable:0 kB > Bounce: 0 kB > CommitLimit: 5468244 kB > Committed_AS: 1946008 kB > VmallocTotal: 253900 kB > VmallocUsed: 2672 kB > VmallocChunk: 251228 kB I assume that meminfo was not captured when the system was ooming? There isn't much slab there. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: 2.6.23-rc8 network problem. Mem leak? ip1000a?
> ntpd. Sounds like pps leaking to me. That's what I'd think, except that pps does no allocation in the normal running state, so there's nothing to leak. The interrupt path just records the time in some preallocated, static buffers and wakes up blocked readers. The read path copies the latest data out of those static buffers. There's allocation when the PPS device is created, and more when it's opened. >> Can anyone offer some diagnosis advice? > CONFIG_DEBUG_SLAB_LEAK? Ah, thanks you; I've been using SLUB which doesn't support this option. Here's what I've extracted. I've only presented the top few slab_allocators and a small subset of the oom-killer messages, but I have full copies if desired. Unfortunately, I've discovered that the machine doesn't live in this unhappy state forever. Indeed, I'm not sure if killing ntpd "fixes" anything; my previous observations may have been optimistic ignorance. (For my own personal reference looking for more oom-kill, I nuked ntpd at 06:46:56. And the oom-kills are continuing, with the latest at 07:43:52.) Anyway, I have a bunch of information from the slab_allocators file, but I'm not quire sure how to make sense of it. With a machine in the unhappy state and firing the OOM killer, the top 20 slab_allocators are: $ sort -rnk2 /proc/slab_allocators | head -20 skbuff_head_cache: 1712746 __alloc_skb+0x31/0x121 size-512: 1706572 tcp_send_ack+0x23/0x102 skbuff_fclone_cache: 149113 __alloc_skb+0x31/0x121 size-2048: 148500 tcp_sendmsg+0x1b5/0xae1 sysfs_dir_cache: 5289 sysfs_new_dirent+0x4b/0xec size-512: 2613 sock_alloc_send_skb+0x93/0x1dd Acpi-Operand: 2014 acpi_ut_allocate_object_desc_dbg+0x34/0x6e size-32: 1995 sysfs_new_dirent+0x29/0xec vm_area_struct: 1679 mmap_region+0x18f/0x421 size-512: 1618 tcp_xmit_probe_skb+0x1f/0xcd size-512: 1571 arp_create+0x4e/0x1cd vm_area_struct: 1544 copy_process+0x9f1/0x1108 anon_vma: 1448 anon_vma_prepare+0x29/0x74 filp: 1201 get_empty_filp+0x44/0xcd UDP: 1173 sk_alloc+0x25/0xaf size-128: 1048 r1bio_pool_alloc+0x23/0x3b size-128: 1024 nfsd_cache_init+0x2d/0xcf Acpi-Namespace: 973 acpi_ns_create_node+0x2c/0x45 vm_area_struct: 717 split_vma+0x33/0xe5 dentry: 594 d_alloc+0x24/0x177 I'm not sure quite what "normal" numbers are, but I do wonder why there are 1.7 million TCP acks buffered in the system. Shouldn't they be transmitted and deallocated pretty quickly? This machine receives more data than it sends, so I'd expect acks to outnumber "real" packets. Could the ip1000a driver's transmit path be leaking skbs somehow? that would also explain the "flailing" of the oom-killer; it can't associate the allocations with a process. Here's /proc/meminfo: MemTotal: 1035756 kB MemFree: 43508 kB Buffers: 72920 kB Cached: 224056 kB SwapCached: 344916 kB Active: 664976 kB Inactive: 267656 kB SwapTotal: 4950368 kB SwapFree: 3729384 kB Dirty:6460 kB Writeback: 0 kB AnonPages: 491708 kB Mapped: 79232 kB Slab:41324 kB SReclaimable:25008 kB SUnreclaim: 16316 kB PageTables: 8132 kB NFS_Unstable:0 kB Bounce: 0 kB CommitLimit: 5468244 kB Committed_AS: 1946008 kB VmallocTotal: 253900 kB VmallocUsed: 2672 kB VmallocChunk: 251228 kB I have a lot of oom-killer messages, that I have saved but am not posting for size reasons, but here are some example backtraces. They're not very helpful to me; do they enlighten anyone else? 02:50:20: apcupsd invoked oom-killer: gfp_mask=0xd0, order=1, oomkilladj=0 02:50:22: 02:50:22: Call Trace: 02:50:22: [] out_of_memory+0x71/0x1ba 02:50:22: [] __alloc_pages+0x255/0x2d7 02:50:22: [] cache_alloc_refill+0x2f4/0x60a 02:50:22: [] hiddev_ioctl+0x579/0x919 02:50:22: [] kmem_cache_alloc+0x57/0x95 02:50:22: [] hiddev_ioctl+0x579/0x919 02:50:22: [] cp_new_stat+0xe5/0xfd 02:50:22: [] hiddev_read+0x199/0x1f6 02:50:22: [] default_wake_function+0x0/0xe 02:50:22: [] do_ioctl+0x45/0x50 02:50:22: [] vfs_ioctl+0x1f9/0x20b 02:50:22: [] sys_ioctl+0x3c/0x5d 02:50:22: [] system_call+0x7e/0x83 02:52:18: postgres invoked oom-killer: gfp_mask=0xd0, order=1, oomkilladj=0 02:52:18: 02:52:18: Call Trace: 02:52:18: [] out_of_memory+0x71/0x1ba 02:52:18: [] __alloc_pages+0x255/0x2d7 02:52:18: [] poison_obj+0x26/0x2f 02:52:18: [] __get_free_pages+0x40/0x79 02:52:18: [] copy_process+0xb0/0x1108 02:52:18: [] alloc_pid+0x1f/0x27d 02:52:18: [] do_fork+0xb1/0x1a7 02:52:18: [] copy_user_generic_string+0x17/0x40 02:52:18: [] system_call+0x7e/0x83 02:52:18: [] ptregscall_common+0x67/0xb0 02:52:18: kthreadd invoked oom-killer: gfp_mask=0xd0, order=1, oomkilladj=0 02:52:18: 02:52:18: Call Trace: 02:52:18: [] out_of_memory+0x71/0x1ba 02:52:18: [] __alloc_pages+0x255/0x2d7 02:52:18: [] __get_free_pages+0x40/0x79 02:52:18: [] copy_process+0xb0/0x1108 02:52:18: [] alloc_pid+0x1f/0x27d 02:52:18: [] do_fork+0xb1/0x1a7 02:52:18: [] update_curr+0xe6/0x10b 02:52:18: []
Re: 2.6.23-rc8 network problem. Mem leak? ip1000a?
ntpd. Sounds like pps leaking to me. That's what I'd think, except that pps does no allocation in the normal running state, so there's nothing to leak. The interrupt path just records the time in some preallocated, static buffers and wakes up blocked readers. The read path copies the latest data out of those static buffers. There's allocation when the PPS device is created, and more when it's opened. Can anyone offer some diagnosis advice? CONFIG_DEBUG_SLAB_LEAK? Ah, thanks you; I've been using SLUB which doesn't support this option. Here's what I've extracted. I've only presented the top few slab_allocators and a small subset of the oom-killer messages, but I have full copies if desired. Unfortunately, I've discovered that the machine doesn't live in this unhappy state forever. Indeed, I'm not sure if killing ntpd fixes anything; my previous observations may have been optimistic ignorance. (For my own personal reference looking for more oom-kill, I nuked ntpd at 06:46:56. And the oom-kills are continuing, with the latest at 07:43:52.) Anyway, I have a bunch of information from the slab_allocators file, but I'm not quire sure how to make sense of it. With a machine in the unhappy state and firing the OOM killer, the top 20 slab_allocators are: $ sort -rnk2 /proc/slab_allocators | head -20 skbuff_head_cache: 1712746 __alloc_skb+0x31/0x121 size-512: 1706572 tcp_send_ack+0x23/0x102 skbuff_fclone_cache: 149113 __alloc_skb+0x31/0x121 size-2048: 148500 tcp_sendmsg+0x1b5/0xae1 sysfs_dir_cache: 5289 sysfs_new_dirent+0x4b/0xec size-512: 2613 sock_alloc_send_skb+0x93/0x1dd Acpi-Operand: 2014 acpi_ut_allocate_object_desc_dbg+0x34/0x6e size-32: 1995 sysfs_new_dirent+0x29/0xec vm_area_struct: 1679 mmap_region+0x18f/0x421 size-512: 1618 tcp_xmit_probe_skb+0x1f/0xcd size-512: 1571 arp_create+0x4e/0x1cd vm_area_struct: 1544 copy_process+0x9f1/0x1108 anon_vma: 1448 anon_vma_prepare+0x29/0x74 filp: 1201 get_empty_filp+0x44/0xcd UDP: 1173 sk_alloc+0x25/0xaf size-128: 1048 r1bio_pool_alloc+0x23/0x3b size-128: 1024 nfsd_cache_init+0x2d/0xcf Acpi-Namespace: 973 acpi_ns_create_node+0x2c/0x45 vm_area_struct: 717 split_vma+0x33/0xe5 dentry: 594 d_alloc+0x24/0x177 I'm not sure quite what normal numbers are, but I do wonder why there are 1.7 million TCP acks buffered in the system. Shouldn't they be transmitted and deallocated pretty quickly? This machine receives more data than it sends, so I'd expect acks to outnumber real packets. Could the ip1000a driver's transmit path be leaking skbs somehow? that would also explain the flailing of the oom-killer; it can't associate the allocations with a process. Here's /proc/meminfo: MemTotal: 1035756 kB MemFree: 43508 kB Buffers: 72920 kB Cached: 224056 kB SwapCached: 344916 kB Active: 664976 kB Inactive: 267656 kB SwapTotal: 4950368 kB SwapFree: 3729384 kB Dirty:6460 kB Writeback: 0 kB AnonPages: 491708 kB Mapped: 79232 kB Slab:41324 kB SReclaimable:25008 kB SUnreclaim: 16316 kB PageTables: 8132 kB NFS_Unstable:0 kB Bounce: 0 kB CommitLimit: 5468244 kB Committed_AS: 1946008 kB VmallocTotal: 253900 kB VmallocUsed: 2672 kB VmallocChunk: 251228 kB I have a lot of oom-killer messages, that I have saved but am not posting for size reasons, but here are some example backtraces. They're not very helpful to me; do they enlighten anyone else? 02:50:20: apcupsd invoked oom-killer: gfp_mask=0xd0, order=1, oomkilladj=0 02:50:22: 02:50:22: Call Trace: 02:50:22: [80246053] out_of_memory+0x71/0x1ba 02:50:22: [8024755d] __alloc_pages+0x255/0x2d7 02:50:22: [8025cbd6] cache_alloc_refill+0x2f4/0x60a 02:50:22: [8040602c] hiddev_ioctl+0x579/0x919 02:50:22: [8025d0fc] kmem_cache_alloc+0x57/0x95 02:50:22: [8040602c] hiddev_ioctl+0x579/0x919 02:50:22: [80262511] cp_new_stat+0xe5/0xfd 02:50:22: [804058ff] hiddev_read+0x199/0x1f6 02:50:22: [80222fa0] default_wake_function+0x0/0xe 02:50:22: [80269bb5] do_ioctl+0x45/0x50 02:50:22: [80269db9] vfs_ioctl+0x1f9/0x20b 02:50:22: [80269e07] sys_ioctl+0x3c/0x5d 02:50:22: [8020b43e] system_call+0x7e/0x83 02:52:18: postgres invoked oom-killer: gfp_mask=0xd0, order=1, oomkilladj=0 02:52:18: 02:52:18: Call Trace: 02:52:18: [80246053] out_of_memory+0x71/0x1ba 02:52:18: [8024755d] __alloc_pages+0x255/0x2d7 02:52:18: [8025be8a] poison_obj+0x26/0x2f 02:52:18: [8024761f] __get_free_pages+0x40/0x79 02:52:18: [80224d66] copy_process+0xb0/0x1108 02:52:18: [80233388] alloc_pid+0x1f/0x27d 02:52:18: [80225ed6] do_fork+0xb1/0x1a7 02:52:18: [802f0627] copy_user_generic_string+0x17/0x40 02:52:18: [8020b43e] system_call+0x7e/0x83 02:52:18: [8020b757] ptregscall_common+0x67/0xb0 02:52:18: kthreadd invoked
Re: 2.6.23-rc8 network problem. Mem leak? ip1000a?
On 30 Sep 2007 03:59:56 -0400 [EMAIL PROTECTED] wrote: ntpd. Sounds like pps leaking to me. That's what I'd think, except that pps does no allocation in the normal running state, so there's nothing to leak. The interrupt path just records the time in some preallocated, static buffers and wakes up blocked readers. The read path copies the latest data out of those static buffers. There's allocation when the PPS device is created, and more when it's opened. OK. Did you try to reproduce it without the pps patch applied? Can anyone offer some diagnosis advice? CONFIG_DEBUG_SLAB_LEAK? Ah, thanks you; I've been using SLUB which doesn't support this option. Here's what I've extracted. I've only presented the top few slab_allocators and a small subset of the oom-killer messages, but I have full copies if desired. Unfortunately, I've discovered that the machine doesn't live in this unhappy state forever. Indeed, I'm not sure if killing ntpd fixes anything; my previous observations may have been optimistic ignorance. (For my own personal reference looking for more oom-kill, I nuked ntpd at 06:46:56. And the oom-kills are continuing, with the latest at 07:43:52.) Anyway, I have a bunch of information from the slab_allocators file, but I'm not quire sure how to make sense of it. With a machine in the unhappy state and firing the OOM killer, the top 20 slab_allocators are: $ sort -rnk2 /proc/slab_allocators | head -20 skbuff_head_cache: 1712746 __alloc_skb+0x31/0x121 size-512: 1706572 tcp_send_ack+0x23/0x102 skbuff_fclone_cache: 149113 __alloc_skb+0x31/0x121 size-2048: 148500 tcp_sendmsg+0x1b5/0xae1 sysfs_dir_cache: 5289 sysfs_new_dirent+0x4b/0xec size-512: 2613 sock_alloc_send_skb+0x93/0x1dd Acpi-Operand: 2014 acpi_ut_allocate_object_desc_dbg+0x34/0x6e size-32: 1995 sysfs_new_dirent+0x29/0xec vm_area_struct: 1679 mmap_region+0x18f/0x421 size-512: 1618 tcp_xmit_probe_skb+0x1f/0xcd size-512: 1571 arp_create+0x4e/0x1cd vm_area_struct: 1544 copy_process+0x9f1/0x1108 anon_vma: 1448 anon_vma_prepare+0x29/0x74 filp: 1201 get_empty_filp+0x44/0xcd UDP: 1173 sk_alloc+0x25/0xaf size-128: 1048 r1bio_pool_alloc+0x23/0x3b size-128: 1024 nfsd_cache_init+0x2d/0xcf Acpi-Namespace: 973 acpi_ns_create_node+0x2c/0x45 vm_area_struct: 717 split_vma+0x33/0xe5 dentry: 594 d_alloc+0x24/0x177 I'm not sure quite what normal numbers are, but I do wonder why there are 1.7 million TCP acks buffered in the system. Shouldn't they be transmitted and deallocated pretty quickly? Yeah, that's an skbuff leak. This machine receives more data than it sends, so I'd expect acks to outnumber real packets. Could the ip1000a driver's transmit path be leaking skbs somehow? Absolutely. Normally a driver's transmit completion interrupt handler will run dev_kfree_skb_irq() against the skbs which have been fully sent. However it'd be darned odd if the driver was leaking only tcp acks. I can find no occurrence of dev_kfree_skb in drivers/net/ipg.c, which is suspicious. Where did you get your ipg.c from, btw? davem's tree? rc8-mm1? rc8-mm2?? that would also explain the flailing of the oom-killer; it can't associate the allocations with a process. Here's /proc/meminfo: MemTotal: 1035756 kB MemFree: 43508 kB Buffers: 72920 kB Cached: 224056 kB SwapCached: 344916 kB Active: 664976 kB Inactive: 267656 kB SwapTotal: 4950368 kB SwapFree: 3729384 kB Dirty:6460 kB Writeback: 0 kB AnonPages: 491708 kB Mapped: 79232 kB Slab:41324 kB SReclaimable:25008 kB SUnreclaim: 16316 kB PageTables: 8132 kB NFS_Unstable:0 kB Bounce: 0 kB CommitLimit: 5468244 kB Committed_AS: 1946008 kB VmallocTotal: 253900 kB VmallocUsed: 2672 kB VmallocChunk: 251228 kB I assume that meminfo was not captured when the system was ooming? There isn't much slab there. - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: 2.6.23-rc8 network problem. Mem leak? ip1000a?
OK. Did you try to reproduce it without the pps patch applied? No. But I've yanked the ip1000a driver (using old crufy vendor-supplied out-of-kernel module) and the problems are GONE. This machine receives more data than it sends, so I'd expect acks to outnumber real packets. Could the ip1000a driver's transmit path be leaking skbs somehow? Absolutely. Normally a driver's transmit completion interrupt handler will run dev_kfree_skb_irq() against the skbs which have been fully sent. However it'd be darned odd if the driver was leaking only tcp acks. It's leaking lots of things... you can see ARP packets in there and all sorts of stuff. But the big traffic hog is BackupPC doing inbound rsyncs all night long, which generates a lot of acks. Those are the packets it sends, so those are the packets that get leaked. I can find no occurrence of dev_kfree_skb in drivers/net/ipg.c, which is suspicious. Look for IPG_DEV_KFREE_SKB, which is a wrapper macro. (Or just add -i to your grep.) It should probably be deleted (it just expands to dev_kfree_skb), but was presumably useful to someone during development. Where did you get your ipg.c from, btw? davem's tree? rc8-mm1? rc8-mm2?? As I wrote originally, I got it from http://marc.info/?l=linux-netdevm=118980588419882 which was a reuqest for mainline submission. If there are other patches floating around, I'm happy to try them. Now that I know what to look for, it's easy to spot the leak before OOM. I assume that meminfo was not captured when the system was ooming? There isn't much slab there. Oops, sorry. I captured slabinfo but not meminfo. Thank you very much! Sorry to jump the gun and post a lot before I had all the data, but if it WAS a problem in -rc8, I wanted to mention it before -final. Now, the rush is to get the ip1000a driver fixed before the merge window opens. I've added all the ip1000a developers to the Cc: list in an attempt to speed that up. - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: 2.6.23-rc8 network problem. Mem leak? ip1000a?
On 27 Sep 2007 22:06:17 -0400 [EMAIL PROTECTED] wrote: > Uniprocessor Althlon 64, 64-bit kernel, 2G ECC RAM, > 2.6.23-rc8 + linuxpps (5.0.0) + ip1000a driver. > (patch from http://marc.info/?l=linux-netdev=118980588419882) > > After a few hours of operation, ntp loses the ability to send packets. > sendto() returns -EAGAIN to everything, including the 24-byte UDP packet > that is a response to ntpq. > > ... > > Killing and restarting ntpd gets it running again for a few hours. > Here's after about two hours of successful operation. (I'll try to > remember to run slabinfo before killing ntpd next time.) ntpd. Sounds like pps leaking to me. > > Can anyone offer some diagnosis advice? > CONFIG_DEBUG_SLAB_LEAK? - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: 2.6.23-rc8 network problem. Mem leak? ip1000a?
On 27 Sep 2007 22:06:17 -0400 [EMAIL PROTECTED] wrote: Uniprocessor Althlon 64, 64-bit kernel, 2G ECC RAM, 2.6.23-rc8 + linuxpps (5.0.0) + ip1000a driver. (patch from http://marc.info/?l=linux-netdevm=118980588419882) After a few hours of operation, ntp loses the ability to send packets. sendto() returns -EAGAIN to everything, including the 24-byte UDP packet that is a response to ntpq. ... Killing and restarting ntpd gets it running again for a few hours. Here's after about two hours of successful operation. (I'll try to remember to run slabinfo before killing ntpd next time.) ntpd. Sounds like pps leaking to me. Can anyone offer some diagnosis advice? CONFIG_DEBUG_SLAB_LEAK? - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
2.6.23-rc8 network problem. Mem leak? ip1000a?
Uniprocessor Althlon 64, 64-bit kernel, 2G ECC RAM, 2.6.23-rc8 + linuxpps (5.0.0) + ip1000a driver. (patch from http://marc.info/?l=linux-netdev=118980588419882) After a few hours of operation, ntp loses the ability to send packets. sendto() returns -EAGAIN to everything, including the 24-byte UDP packet that is a response to ntpq. -EAGAIN on a sendto() makes me think of memory problems, so here's meminfo at the time: ### FAILED state ### # cat /proc/meminfo MemTotal: 2059384 kB MemFree: 15332 kB Buffers:665608 kB Cached: 18212 kB SwapCached: 0 kB Active: 380384 kB Inactive: 355020 kB SwapTotal: 5855208 kB SwapFree: 5854552 kB Dirty: 28504 kB Writeback: 0 kB AnonPages: 51608 kB Mapped: 11852 kB Slab: 1285348 kB SReclaimable: 152968 kB SUnreclaim:1132380 kB PageTables: 3888 kB NFS_Unstable:0 kB Bounce: 0 kB CommitLimit: 6884900 kB Committed_AS: 590528 kB VmallocTotal: 34359738367 kB VmallocUsed:265628 kB VmallocChunk: 34359472059 kB Killing and restarting ntpd gets it running again for a few hours. Here's after about two hours of successful operation. (I'll try to remember to run slabinfo before killing ntpd next time.) ### WORKING state ### # cat /proc/meminfo MemTotal: 2059384 kB MemFree: 20252 kB Buffers:242688 kB Cached: 41556 kB SwapCached:200 kB Active: 285012 kB Inactive: 147348 kB SwapTotal: 5855208 kB SwapFree: 5854212 kB Dirty: 36 kB Writeback: 0 kB AnonPages: 148052 kB Mapped: 12756 kB Slab: 1582512 kB SReclaimable: 134348 kB SUnreclaim:1448164 kB PageTables: 4500 kB NFS_Unstable:0 kB Bounce: 0 kB CommitLimit: 6884900 kB Committed_AS: 689956 kB VmallocTotal: 34359738367 kB VmallocUsed:265628 kB VmallocChunk: 34359472059 kB # /usr/src/linux/Documentation/vm/slabinfo Name Objects ObjsizeSpace Slabs/Part/Cpu O/S O %Fr %Ef Flg :016 1478 1624.5K 6/3/1 256 0 50 96 * :024 170 24 4.0K 1/0/1 170 0 0 99 * :032 1339 3245.0K 11/2/1 128 0 18 95 * :040 102 40 4.0K 1/0/1 102 0 0 99 * :064 5937 64 413.6K 101/15/1 64 0 14 91 * :07256 72 4.0K 1/0/1 56 0 0 98 * :088 6946 88 618.4K151/0/1 46 0 0 98 * :096 23851 96 2.5M 616/144/1 42 0 23 90 * :128 730 128 114.6K 28/6/1 32 0 21 81 * :136 232 13636.8K 9/6/1 30 0 66 85 * :192 474 19298.3K 24/4/1 21 0 16 92 * :256 1385376 256 354.6M 86587/0/1 16 0 0 99 * :32012 304 4.0K 1/0/1 12 0 0 89 *A :384 359 384 180.2K44/23/1 10 0 52 76 *A :512 1384316 512 708.7M 173040/1/18 0 0 99 * :64072 61653.2K 13/5/16 0 38 83 *A :704 1870 696 1.3M170/0/1 11 1 0 93 *A :0001024 4271024 454.6K111/9/14 0 8 96 * :0001472 1501472 245.7K 30/0/15 1 0 89 * :00020481589912048 325.7M 39759/25/14 1 0 99 * :0004096514096 245.7K 30/9/12 1 30 85 * Acpi-State 51 80 4.0K 1/0/1 51 0 0 99 anon_vma 1032 1628.6K 7/5/1 170 0 71 57 bdev_cache 43 72036.8K 9/1/15 0 11 83 Aa blkdev_requests 42 28812.2K 3/0/1 14 0 0 98 buffer_head 59173 10411.1M2734/1690/1 39 0 61 54 a cfq_io_context 223 15240.9K 10/6/1 26 0 60 82 dentry 98641 19219.7M 4813/274/1 21 0 5 96 a ext3_inode_cache115690 68886.3M 10545/77/1 11 1 0 92 a file_lock_cache 23 168 4.0K 1/0/1 23 0 0 94 idr_layer_cache118 52869.6K 17/1/17 0 5 89 inode_cache 1365 528 798.7K195/0/17 0 0 90 a kmalloc-131072 1 131072 131.0K 1/0/11 5 0 100 kmalloc-163848 16384 131.0K 8/0/11 2 0 100 kmalloc-327681 3276832.7K 1/0/11 3 0 100 kmalloc-8 1535 812.2K 3/1/1 512 0 33 99 kmalloc-819210
2.6.23-rc8 network problem. Mem leak? ip1000a?
Uniprocessor Althlon 64, 64-bit kernel, 2G ECC RAM, 2.6.23-rc8 + linuxpps (5.0.0) + ip1000a driver. (patch from http://marc.info/?l=linux-netdevm=118980588419882) After a few hours of operation, ntp loses the ability to send packets. sendto() returns -EAGAIN to everything, including the 24-byte UDP packet that is a response to ntpq. -EAGAIN on a sendto() makes me think of memory problems, so here's meminfo at the time: ### FAILED state ### # cat /proc/meminfo MemTotal: 2059384 kB MemFree: 15332 kB Buffers:665608 kB Cached: 18212 kB SwapCached: 0 kB Active: 380384 kB Inactive: 355020 kB SwapTotal: 5855208 kB SwapFree: 5854552 kB Dirty: 28504 kB Writeback: 0 kB AnonPages: 51608 kB Mapped: 11852 kB Slab: 1285348 kB SReclaimable: 152968 kB SUnreclaim:1132380 kB PageTables: 3888 kB NFS_Unstable:0 kB Bounce: 0 kB CommitLimit: 6884900 kB Committed_AS: 590528 kB VmallocTotal: 34359738367 kB VmallocUsed:265628 kB VmallocChunk: 34359472059 kB Killing and restarting ntpd gets it running again for a few hours. Here's after about two hours of successful operation. (I'll try to remember to run slabinfo before killing ntpd next time.) ### WORKING state ### # cat /proc/meminfo MemTotal: 2059384 kB MemFree: 20252 kB Buffers:242688 kB Cached: 41556 kB SwapCached:200 kB Active: 285012 kB Inactive: 147348 kB SwapTotal: 5855208 kB SwapFree: 5854212 kB Dirty: 36 kB Writeback: 0 kB AnonPages: 148052 kB Mapped: 12756 kB Slab: 1582512 kB SReclaimable: 134348 kB SUnreclaim:1448164 kB PageTables: 4500 kB NFS_Unstable:0 kB Bounce: 0 kB CommitLimit: 6884900 kB Committed_AS: 689956 kB VmallocTotal: 34359738367 kB VmallocUsed:265628 kB VmallocChunk: 34359472059 kB # /usr/src/linux/Documentation/vm/slabinfo Name Objects ObjsizeSpace Slabs/Part/Cpu O/S O %Fr %Ef Flg :016 1478 1624.5K 6/3/1 256 0 50 96 * :024 170 24 4.0K 1/0/1 170 0 0 99 * :032 1339 3245.0K 11/2/1 128 0 18 95 * :040 102 40 4.0K 1/0/1 102 0 0 99 * :064 5937 64 413.6K 101/15/1 64 0 14 91 * :07256 72 4.0K 1/0/1 56 0 0 98 * :088 6946 88 618.4K151/0/1 46 0 0 98 * :096 23851 96 2.5M 616/144/1 42 0 23 90 * :128 730 128 114.6K 28/6/1 32 0 21 81 * :136 232 13636.8K 9/6/1 30 0 66 85 * :192 474 19298.3K 24/4/1 21 0 16 92 * :256 1385376 256 354.6M 86587/0/1 16 0 0 99 * :32012 304 4.0K 1/0/1 12 0 0 89 *A :384 359 384 180.2K44/23/1 10 0 52 76 *A :512 1384316 512 708.7M 173040/1/18 0 0 99 * :64072 61653.2K 13/5/16 0 38 83 *A :704 1870 696 1.3M170/0/1 11 1 0 93 *A :0001024 4271024 454.6K111/9/14 0 8 96 * :0001472 1501472 245.7K 30/0/15 1 0 89 * :00020481589912048 325.7M 39759/25/14 1 0 99 * :0004096514096 245.7K 30/9/12 1 30 85 * Acpi-State 51 80 4.0K 1/0/1 51 0 0 99 anon_vma 1032 1628.6K 7/5/1 170 0 71 57 bdev_cache 43 72036.8K 9/1/15 0 11 83 Aa blkdev_requests 42 28812.2K 3/0/1 14 0 0 98 buffer_head 59173 10411.1M2734/1690/1 39 0 61 54 a cfq_io_context 223 15240.9K 10/6/1 26 0 60 82 dentry 98641 19219.7M 4813/274/1 21 0 5 96 a ext3_inode_cache115690 68886.3M 10545/77/1 11 1 0 92 a file_lock_cache 23 168 4.0K 1/0/1 23 0 0 94 idr_layer_cache118 52869.6K 17/1/17 0 5 89 inode_cache 1365 528 798.7K195/0/17 0 0 90 a kmalloc-131072 1 131072 131.0K 1/0/11 5 0 100 kmalloc-163848 16384 131.0K 8/0/11 2 0 100 kmalloc-327681 3276832.7K 1/0/11 3 0 100 kmalloc-8 1535 812.2K 3/1/1 512 0 33 99 kmalloc-819210
