Re: [ccp4bb] Advice on setting up / maintaining a Ubuntu cluster
I wanted to thank everyone who responded, for a whole bunch of advice and suggestions! Sergei On 29-Jul-13 12:22 PM, Sergei Strelkov wrote: Dear all, In old times I, just like about any protein crystallographer, used to work on a cluster of SGI/IRIX workstations with complete NFS-based cross-mounting of hard disks. A typical operation included: 1. A single home directory location for every user: if my home directory was on workstation X, I would by default use it after logging on any of the workstations in the cluster. 2. A single location for all software for general use. (And, obviously, 3. The ability to log on any node from any terminal; today this is done via the 'ssh -X' command). I wondered if someone could give us an advice on a painless setup enabling 1. and 2., for a small cluster of Ubuntu computers. We (will) have about five similar Dell computers in a local (192.168.*.*) network (wired/wireless). Any tips on the hardware (especially the LAN and network disks) are also welcome. Many thanks, Sergei -- Prof. Sergei V. Strelkov Laboratory for Biocrystallography Dept of Pharmaceutical and Pharmacological Sciences, KU Leuven Herestraat 49 bus 822, 3000 Leuven, Belgium Work phone: +32 16 330845 Mobile: +32 486 294132 Lab pages: http://pharm.kuleuven.be/anafar
Re: [ccp4bb] Advice on setting up / maintaining a Ubuntu cluster
Dear Sergei, Many have already given good advice. A somewhat different approach: You might want to consider using a NAS for user file and program storage. We have run a group of Linux PCs for years that way and we are quite pleased with it. The NAS would take the place of your file server PC. Although I don't quite have enough experience with it yet to fully recommend it, I do think you might want to consider using FreeNAS (just Google it), rather than buying a physical NAS. In a nutshell: 1) It is a free NAS setup, as the name suggests. 2) You need a PC with disk bays, a fair CPU and LOTS of memory. Add number and size of the disks you want. The PC generally should be a 64-bit system or else you will not be able to install enough memory. 3) You download the operating system on a USB drive (it does not go on your data disks) - this is very easy to do 4) Follow setup instructions and you are ready to start testing. FreeNAS offers many options, such as mirroring, striping etc. Different kinds of RAIDS are possible. My experience has been that it is very nice to have a RAID system that allows for one disk to go bad and you still have a working system. It has happened 2-3 times in 5 years, on 3 NASes combined, that we have had a disk fail without notice. The FreeNAS home page will lead you to all the RAID options you have, how safe they are, what the read and write performance is etc. It is a good idea to build a second system in a different building for backup. We use Rsync for backup at night and this works well (it is slow but it does not matter). You might want to do a resource analysis what your limiting factor is in processing crystallographic data. My experience is that the network generally is not the limiting factor and disk access generally is not the limit either - but it can be, if you are transferring a large number of images from a synchrotron. Typically, in my experience, CPU is still the limiting factor. Hope this helps. Mark -Original Message- From: Sergei Strelkov sergei.strel...@pharm.kuleuven.be To: CCP4BB CCP4BB@JISCMAIL.AC.UK Sent: Mon, Jul 29, 2013 4:22 am Subject: [ccp4bb] Advice on setting up / maintaining a Ubuntu cluster Dear all, In old times I, just like about any protein crystallographer, used to work on a cluster of SGI/IRIX workstations with complete NFS-based cross-mounting of hard disks. A typical operation included: 1. A single home directory location for every user: if my home directory was on workstation X, I would by default use it after logging on any of the workstations in the cluster. 2. A single location for all software for general use. (And, obviously, 3. The ability to log on any node from any terminal; today this is done via the 'ssh -X' command). I wondered if someone could give us an advice on a painless setup enabling 1. and 2., for a small cluster of Ubuntu computers. We (will) have about five similar Dell computers in a local (192.168.*.*) network (wired/wireless). Any tips on the hardware (especially the LAN and network disks) are also welcome. Many thanks, Sergei -- Prof. Sergei V. Strelkov Laboratory for Biocrystallography Dept of Pharmaceutical and Pharmacological Sciences, KU Leuven Herestraat 49 bus 822, 3000 Leuven, Belgium Work phone: +32 16 330845 Mobile: +32 486 294132 Lab pages: http://pharm.kuleuven.be/anafar
[ccp4bb] Advice on setting up / maintaining a Ubuntu cluster
Dear all, In old times I, just like about any protein crystallographer, used to work on a cluster of SGI/IRIX workstations with complete NFS-based cross-mounting of hard disks. A typical operation included: 1. A single home directory location for every user: if my home directory was on workstation X, I would by default use it after logging on any of the workstations in the cluster. 2. A single location for all software for general use. (And, obviously, 3. The ability to log on any node from any terminal; today this is done via the 'ssh -X' command). I wondered if someone could give us an advice on a painless setup enabling 1. and 2., for a small cluster of Ubuntu computers. We (will) have about five similar Dell computers in a local (192.168.*.*) network (wired/wireless). Any tips on the hardware (especially the LAN and network disks) are also welcome. Many thanks, Sergei -- Prof. Sergei V. Strelkov Laboratory for Biocrystallography Dept of Pharmaceutical and Pharmacological Sciences, KU Leuven Herestraat 49 bus 822, 3000 Leuven, Belgium Work phone: +32 16 330845 Mobile: +32 486 294132 Lab pages: http://pharm.kuleuven.be/anafar
Re: [ccp4bb] Advice on setting up / maintaining a Ubuntu cluster
Dear Sergei, IMO, the easiest way to achieve your goals is good old NIS and NFS with a centralized server on wired gigabit network. You could go with LDAP instead of NIS, but it is considerably more difficult to set up. One computer would act as a server, containing the user database, homes and programs. Hardware RAID is not worth it. You are better off getting a Linux-supported SAS/SATA HBA (e.g. Dell SAS 6/iR) and making a software RAID 5 with mdadm out of a bunch of inexpensive consumer-grade SATA disks. You need a minimum of 4 drives for RAID5. An external HDD enclosure might be necessary depending on server's chassis and the desired number of drives. We built our server from an old P4 workstation with a couple gigs of RAM (8 clients). Having two or more cores is a benefit. If I am not mistaken, software RAID 5 is not bootable, so you would need an extra drive (can be very small) for the core part of the OS. Export /home and /usr/local with NFS, mount them from client machines, hook the clients up to NIS and you are done. Some programs might not reside in /usr/local in which case you would have to export and mount more directories. Ubuntu community has pretty good and easy to follow guides for NIS, NFS and mdadm. Bets regards, Dmitry On 2013-07-29, at 6:22 AM, Sergei Strelkov wrote: Dear all, In old times I, just like about any protein crystallographer, used to work on a cluster of SGI/IRIX workstations with complete NFS-based cross-mounting of hard disks. A typical operation included: 1. A single home directory location for every user: if my home directory was on workstation X, I would by default use it after logging on any of the workstations in the cluster. 2. A single location for all software for general use. (And, obviously, 3. The ability to log on any node from any terminal; today this is done via the 'ssh -X' command). I wondered if someone could give us an advice on a painless setup enabling 1. and 2., for a small cluster of Ubuntu computers. We (will) have about five similar Dell computers in a local (192.168.*.*) network (wired/wireless). Any tips on the hardware (especially the LAN and network disks) are also welcome. Many thanks, Sergei -- Prof. Sergei V. Strelkov Laboratory for Biocrystallography Dept of Pharmaceutical and Pharmacological Sciences, KU Leuven Herestraat 49 bus 822, 3000 Leuven, Belgium Work phone: +32 16 330845 Mobile: +32 486 294132 Lab pages: http://pharm.kuleuven.be/anafar smime.p7s Description: S/MIME cryptographic signature
