Ivan Dichev a écrit :
Arnaldo Carvalho de Melo wrote:
Em Fri, Jan 25, 2008 at 02:21:08PM +0100, Andi Kleen escreveu:
"Ivan H. Dichev" <[EMAIL PROTECTED]> writes:
What could happen if I put different Lan card in every slot?
In ex. to-private -> 3com
to-inet -> VIA
to-dmz -> rtl8139
And then to look which RX function is consuming the memory.
(boomerang_rx, rtl8139_rx, ... etc)
The problem is unlikely to be in the driver (these are both
well tested ones) but more likely your complicated iptables setup somehow
triggers a skb leak.
There are unfortunately no shrink wrapped debug mechanisms in the kernel
for leaks like this (ok you could enable CONFIG_NETFILTER_DEBUG
and see if it prints something interesting, but that's a long shot).
If you wanted to write a custom debugging patch I would do something like this:
- Add two new integer fields to struct sk_buff: a time stamp and a integer field
- Fill the time stamp with jiffies in alloc_skb and clear the integer field
- In __kfree_skb clear the time stamp
- For all the ipt target modules in net/ipv4/netfilter/*.c you use change their
->target functions to put an unique value into the integer field you added.
- Do the same for the pkt_to_tuple functions for all conntrack modules
Then when you observe the leak take a crash dump using kdump on the router
and then use crash to dump all the slab objects for the sk_head_cache.
Then look for any that have an old time stamp and check what value they
have in the integer field. Then the netfilter function who set that unique value
likely triggered the leak somehow.
I wrote some systemtap scripts that do parts of what you suggest, and at
least for the timestamp there was no need to add a new field to struct
sk_buff, I just reuse skb->timestamp, as it is only used when we use a
packet sniffer. Here it is for reference, but it needs some tapsets I
wrote, so I'll publish this git repo in git.kernel.org, perhaps it can
be useful in this case as a starting point. Find another unused field
(hint: I know that at least 4 bytes on 64 bits is present as a hole) and
you're done, no need to rebuild the kernel :)
http://git.kernel.org/?p=linux/kernel/git/acme/nettaps.git
- Arnaldo
Thanks to everyone for the given ideas.
I am not kernel guru so writing patch is difficult. This is a production
server and it is quite difficult to debug (only at night)
I removed some iptables exotics - recent , ulog, string , but no effect.
Since we can reach OOM most of the memory is going to be filled with the
leak, and we are thinking to try to dump and analyze it.
We have looked at the "crash" tool, and we will see what we can do with
it. Meanwhile do you have any hint/ideas ?
Thanks a lot.
I understand you dont want to tell us exact firewall rules you have.
Maybe you could post at least following infos :
# cat /proc/slabinfo
# lsmod
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at http://vger.kernel.org/majordomo-info.html
