Re: Out of socket memory? (2.4.0-test11)
Dan Kegel wrote:
> Daniel Walton ([EMAIL PROTECTED]) wrote:
> > I've been having a problem with a high volume Linux web server. This
> > particular web server used to be a FreeBSD machine and I've been trying to
> > successfully make the switch for some time now. I've been trying the 2.4
> > development kernels as they come out and I've been tweaking the /proc
> > filesystem variables but so far nothing seems to have fixed the
> > problem. The problem is that I get "Out of socket memory" errors and the
> > networking locks up...
>
> FWIW, the code that prints this is in ipv4/tcp_timer.c
> and it looks like it's a DoS attack countermeasure.
>
> /* Do not allow orphaned sockets to eat all our resources.
> * This is direct violation of TCP specs, but it is required
> * to prevent DoS attacks. It is called when a retransmission timeout
> * or zero probe timeout occurs on orphaned socket.
> ...
> if (orphans >= sysctl_tcp_max_orphans ||
> (sk->wmem_queued > SOCK_MIN_SNDBUF &&
> atomic_read(&tcp_memory_allocated) > sysctl_tcp_mem[2])) {
> if (net_ratelimit())
> printk(KERN_INFO "Out of socket memory\n");
See Documentation/networking/ip-sysctl.txt for more info and tuning
suggestions, esp. tcp_orphan_retries and tcp_max_orphans.
An orphaned socket is one that has at least partially
opened, but has not been fully accepted. You might try reducing
tcp_orphan_retries.
It looks like you can view the current number of orphans at
/proc/net/sockstat or something like that.
/*
* Report socket allocation statistics [[EMAIL PROTECTED]]
*/
int afinet_get_info(char *buffer, char **start, off_t offset, int length)
{
/* From net/socket.c */
extern int socket_get_info(char *, char **, off_t, int);
int len = socket_get_info(buffer,start,offset,length);
len += sprintf(buffer+len,"TCP: inuse %d orphan %d tw %d alloc %d mem %d\n",
fold_prot_inuse(&tcp_prot),
atomic_read(&tcp_orphan_count), tcp_tw_count,
atomic_read(&tcp_sockets_allocated),
atomic_read(&tcp_memory_allocated));
- Dan
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/
Re: Out of socket memory? (2.4.0-test11)
Hello! > What am I doing wrong? You change parameters without investigating why failure happened. This approach cannot succeed, of course. > problem? Is there any way I can get runtime information from the kernel on > things like amount of socket memory used and amount available? cat /proc/net/sockstat cat /proc/net/netstat cat /proc/net/snmp cat /proc/slabinfo cat /proc/net/tcp Next time when you will see these messages, do this, pack the result and send to me pointopoint. Alexey - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: Out of socket memory? (2.4.0-test11)
On Wed, Dec 06, 2000 at 10:56:32PM -0600, Daniel Walton wrote: > > Hello, > > I've been having a problem with a high volume Linux web server. This > particular web server used to be a FreeBSD machine and I've been trying to > successfully make the switch for some time now. I've been trying the 2.4 > development kernels as they come out and I've been tweaking the /proc > filesystem variables but so far nothing seems to have fixed the > problem. The problem is that I get "Out of socket memory" errors and the > networking locks up. Sometimes the server will go for weeks without You should probably first find out what and why is generating these messages. It isn't the kernel. When the web server is generating when a send blocks then it is badly broken. -Andi - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: Out of socket memory? (2.4.0-test11)
Daniel Walton ([EMAIL PROTECTED]) wrote:
> I've been having a problem with a high volume Linux web server. This
> particular web server used to be a FreeBSD machine and I've been trying to
> successfully make the switch for some time now. I've been trying the 2.4
> development kernels as they come out and I've been tweaking the /proc
> filesystem variables but so far nothing seems to have fixed the
> problem. The problem is that I get "Out of socket memory" errors and the
> networking locks up...
FWIW, the code that prints this is in ipv4/tcp_timer.c
and it looks like it's a DoS attack countermeasure.
Looks like you can tune it with sysctl_tcp_max_orphans, among
other things. Here's the code from test11-pre4:
/* Do not allow orphaned sockets to eat all our resources.
* This is direct violation of TCP specs, but it is required
* to prevent DoS attacks. It is called when a retransmission timeout
* or zero probe timeout occurs on orphaned socket.
*
* Criterium is still not confirmed experimentally and may change.
* We kill the socket, if:
* 1. If number of orphaned sockets exceeds an administratively configured
*limit.
* 2. If we have strong memory pressure.
*/
static int tcp_out_of_resources(struct sock *sk, int do_reset)
{
struct tcp_opt *tp = &(sk->tp_pinfo.af_tcp);
int orphans = atomic_read(&tcp_orphan_count);
/* If peer does not open window for long time, or did not transmit
* anything for long time, penalize it. */
if ((s32)(tcp_time_stamp - tp->lsndtime) > 2*TCP_RTO_MAX || !do_reset)
orphans <<= 1;
/* If some dubious ICMP arrived, penalize even more. */
if (sk->err_soft)
orphans <<= 1;
if (orphans >= sysctl_tcp_max_orphans ||
(sk->wmem_queued > SOCK_MIN_SNDBUF &&
atomic_read(&tcp_memory_allocated) > sysctl_tcp_mem[2])) {
if (net_ratelimit())
printk(KERN_INFO "Out of socket memory\n");
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
Please read the FAQ at http://www.tux.org/lkml/
Re: Out of socket memory? (2.4.0-test11)
I'm not quite clear how the settings under /proc/sys/vm/* would effect the problem. I neglected to mention in my previous post that all web content is served directly from the memory of the web server (no file accesses). The only file accesses that happen are from a MySQL server which gets queried about once a second. Here's the output of /proc/meminfo. I'm not sure how helpful it is. I was kinda hoping for something that would allow me to see how much memory had been allocated for sockets and what the max was. [root@s4 /proc]# cat meminfo total:used:free: shared: buffers: cached: Mem: 261742592 122847232 1388953600 1757184 88633344 Swap: 2713927680 271392768 MemTotal: 255608 kB MemFree:135640 kB MemShared: 0 kB Buffers: 1716 kB Cached: 86556 kB Active: 15684 kB Inact_dirty: 72588 kB Inact_clean: 0 kB Inact_target: 68 kB HighTotal: 0 kB HighFree:0 kB LowTotal: 255608 kB LowFree:135640 kB SwapTotal: 265032 kB SwapFree: 265032 kB -Dan At 12:30 AM 12/7/2000 -0500, you wrote: >backlog queue? tuning /proc/sys/vm/*? > > > problem? Is there any way I can get runtime information from the > kernel on > > things like amount of socket memory used and amount available? Am I using > >/proc/meminfo? - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Out of socket memory? (2.4.0-test11)
Hello, I've been having a problem with a high volume Linux web server. This particular web server used to be a FreeBSD machine and I've been trying to successfully make the switch for some time now. I've been trying the 2.4 development kernels as they come out and I've been tweaking the /proc filesystem variables but so far nothing seems to have fixed the problem. The problem is that I get "Out of socket memory" errors and the networking locks up. Sometimes the server will go for weeks without running into the problem and other times it'll last 30 minutes. The hardware in question is an 1Ghz Athalon system with 256Mb of ram and an IDE hard disk. I've tried every 2.4 test kernel to date. The web server is a specialized web server running about 10 million hits a day. Of the 256Mb of ram the web server uses 40Mb and there are no other significant memory consuming processes on the system. Currently I am using the following /proc modifications in the rc.local file. echo "7168 11776 16384" > /proc/sys/net/ipv4/tcp_mem echo 32768 > /proc/sys/net/ipv4/tcp_max_orphans What am I doing wrong? Is this a kernel problem or a configuration problem? Is there any way I can get runtime information from the kernel on things like amount of socket memory used and amount available? Am I using the right variables to increase available socket memory and just not giving it enough yet? I appreciate any help provided. Thank you, Daniel Walton - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
