On Sat, 2006-05-06 at 15:09 +0200, Loic Dachary wrote:
> Hi,
>
> After the GNA machines are moved to the new location, they will
> be plugged on a gigabit link. The cumulated bandwidth of all machines
> on our switch can occasionaly require more than 100Mbs (as little as
> 10 people downloading a large file at the same time from an ADSL2+
> line for instance).
>
> I occasionaly noticed that lisa.gna.org was not serving as much
> as it could, even when its load average is low. Anyway, nowadays 100Mbs is
> just not enough when millions of developers have 10Mbs in download.
>
> I'll buy a gigabit switch and a few gigabit ethernet cards.
> I'll be thinking about the ratio between maximum bandwidth (100MB/s) and
> the average capacity of a IDE/SATA drive (between ~20MB/s and 40MB/s
> and 20MB/s is a safe conservative number.
>
> It may be a good idea to have a SCSI machine for serving data
> since the limit is rack space and power. Let me know what you think.
I've made some experiments with file servers for about a year (we
build backup servers for SMEs), and had some interesting experience with
SATA.
One has to carefully choose the SATA controler: 3Ware PCI cards are
quite unexpensive and work fare more reliably than any mobo embedded
controler. Each SATA channel is independent and does provide a very nice
bandwidth (we easily have 50MB/s reads with off-the-shelf disks).
RAID also works fine. With software raid, I never managed to exceed a
0.5 load on a 2Ghz server. The CPU to IO cycles ratio is very high
nowadays, and specialized processors are not really necessary,
especially if your server is dedicated to I/O. I also found one RAID
SATA controler which worked nice, only tested with RAID1. The
'linuxraid' project should guarantee that we can still access the RAID
array via software if the hardware had to be swapped.
Note that most SATA controlers are branded as 'RAID' but really aren't
(ex: DELL ones), they just embed a hash/parity coprocessor and Linux
does not care at all about it. Hard RAID also requires to either reboot
and go BIOS to control/monitor the array, or use proprietary tools
('mdadm' daemon becomes useless, sigh).
The last problem we're having is cooling. Buildind a 3-disks RAID5
server is currently far better off with a big ugly tower. DELL (and
other taiwanese clones) have slick 1U 3-disk-slots machines, but I'm
doubtful about their cooling.
Summing up, today I praise for 3x250GB SATA soft RAID5 servers, with
carefully choosen controlers. Cheap, efficient, reliable.
[BTW: I mean 50MB/s disk I/O, that's 400Mbps network I/O]
As for Gna!: currently, Lisa hosts cvs+svn+arch+download. Moving
storage to a dedicated server (can be done gradually) won't change this
single point of failure. But it would be more reliable (RAID), and we
would separate (and eliminate for now) CPU load and I/O load issues. And
we already use NFS all other the place.
Oh, and we also would need as many RAM as we could. Each service (svn,
cvs, etc) access a specific part of the disk and has its own usage
pattern. This gives a lot of scattered access and decrease I/O bandwidth
very fast. Lots of RAM improves locality and helps Linux bulks its
accesses more efficiently (with a little tuning).
_______________________________________________
Project mailing list
[email protected]
http://mail.gna.org:8080/listinfo/project
