Initial test setup: two machines, running connections between them.
Mostly asymetric (about 50Mbps in one direction,
GigE in the other). Each connection is trying some random rate between 128kbps
and 3Mbps in one direction, and 1kbps in the other direction.
Sending machine is dual 3.0Ghz xeons, 1MB cache, HT, and emt64 (running 32-bit
kernel & user space though). 1GB of RAM
Receiving machine is dual 2.8Ghz xeons, 512 MB cache, HT, 32-bit. 2GB of RAM
(but only 850Mbps of low memory of course...saw the thing OOM kill me with 1GB of
free high memory :( )
Zero latency:
2000 TCP connections: When I first start, I see errors indicating I'm out of low
memory..but it quickly recovers. Probably because my program takes a small
bit of time before it starts reading the sockets.
986Mbps of ethernet traffic (counting all ethernet headers)
3000 TCP connections: Same memory issue 986Mbps of ethernet traffic, about 82kpps
4000 TCP connections: Had to drop max_backlog to 5000 from 10000 to keep the machine from going OOM and killing my traffic generator (on the receiving side). 986Mbps of ethernet traffic
I will work on some numbers with latency tomorrow (had to stop and re-write some of my code to better handle managing the 8000 endpoints that 4000 connections requires!)
I think we can assume that the problem is either related to latency, or sendfile, since 4000 connections with no latency rocks along just fine...
Hmmmm.... can you try to following just to exclude some theories:
Run it with 4000 sockets and then do the following on the server-machine:
dd if=/dev/zero of=file1 bs=1M count=1024 dd if=/dev/zero of=file2 bs=1M count=1024 dd if=/dev/zero of=file3 bs=1M count=1024 cat file1 > /dev/zero & cat file2 > /dev/zero & cat file3 > /dev/zero &
I THINK it might have something to do with caching-pressure or so. See if there is a slow-down on the sending if the page-cache gets full and has to be cleared again.
You are running 2.6.11?
Chris - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
