Ahh, there is nothing quite like a nice big cluster to bring any file
server to its knees.
My experience with cases like this is that the culprit is usually NFS,
or the disk file system being used on the RAID array. It is generally
NOT a bandwidth problem. Basically, I suspect the bottleneck is the
kernel on your file server trying to figure out where to put all these
blocks that are flying in from the cluster. One of the most expensive
things you can do to a file server is an NFS write. This is the reason
why NFS has the "noatime" option, since reading a file normally involves
updating the "access time" for that file (and this is a "write"
operation). Alternately, the disk file system itself (ext3?) can also
get bogged down with many simultaneous writes. XFS is supposed to be
less prone to this problem, but I have heard mixed reviews. In
addition, writing a large number of files simultaneously seems to be a
great way to fragment your file system. I don't know why this is so,
but I once used it as a protocol to create a small, heavily fragmented
file system for testing purposes! If the file system is fragmented,
then access to it just starts getting slow. No warnings, no CPU
utilization, just really really long delays to get an "ls".
That is my guess. What I suggest doing is to come up with a series of
tests that "simulate" this event at various stages. That is, first use
the timeless classic unix "dd" command to generate a crunch of 2GB files
pseudo-simultaneously LOCALLY on the file server:
/bin/tcsh
set time
foreach file ( 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 )
dd if=/dev/zero bs=2G count=1 of=/home/username/deleteme$file
end
If this hoses the home directories, then you know the cluster and
network has nothing to do with your problem. You can do a
quick-and-dirty benchmark of your RAID performance like this too.
Setting the "time" variable above means you will get timing statistics
for each command. If you divide the total number of bytes by the total
time, then you get the average write performance. It will be
interesting to play with the number of files as well as the division of
the 2GB into blocks with the "bs" and "count" options. Large block
sizes will eat a lot of memory and small block sizes will eat a lot of
CPU. Setting your NFS block size to the maximum (in /etc/fstab) is
generally a good idea for scientific computing. Also, network blocks
are typically 1500 bytes (the MTU size). If this block size is a
problem, then you might want to consider "jumbo frames" on your cluster
subnet.
However, I expect the best thing is to find a way to avoid a large
number of simultaneous NFS writes. Either use a different transfer
protocol (such as rcp or nc), or use some kind of "lock file" to prevent
your completing jobs from copying their files all at the same time.
HTH
-James Holton
MAD Scientist
Harry M. Greenblatt wrote:
BS"D
To those hardware oriented:
We have a compute cluster with 23 nodes (dual socket, dual core
Intel servers). Users run simulation jobs on the nodes from the head
node. At the end of each simulation, a result file is compressed to
2GB, and copied to the file server for the cluster (not the head node)
via NFS. Each node is connected via a Gigabit line to a switch. The
file server has a 4-link aggregated Ethernet trunk (4Gb/S) to the
switch. The file server also has two sockets, with Dual Core Xeon
2.1GHz CPU's and 4 GB of memory, running RH4. There are two raid
arrays (RAID 5), each consisting of 8x500GB SATA II WD server drives,
with one file system on each. The raid cards are AMCC 3WARE 9550 and
9650SE (PCI-Express) with 256 MB of cache memory .
When several (~10) jobs finish at once, and the nodes start copying
the compressed file to the file server, the load on the file server
gets very high (~10), and the users whose home directory are on the
file server cannot work at their stations. Using nmon to locate the
bottleneck, it appears that disk I/O is the problem. But the numbers
being reported are a bit strange. It reports a throughput of only
about 50MB/s, and claims the "disk" is 100% busy. These raid cards
should give throughput in the several hundred MB/s range, especially
the 9650 which is rated at 600MB/s RAID 6 write (and we have RAID 5).
1) Is there a more friendly system load monitoring tool we can use?
2) The users may be able to stagger the output schedule of their
jobs, but based on the numbers, we get the feeling the RAID arrays are
not performing as they should. Any suggestions?
Thanks
Harry
-------------------------------------------------------------------------
Harry M. Greenblatt
Staff Scientist
Dept of Structural Biology [EMAIL PROTECTED]
<mailto:[EMAIL PROTECTED]>
Weizmann Institute of Science Phone: 972-8-934-3625
Rehovot, 76100 Facsimile: 972-8-934-4159
Israel
