My blog is here: http://www.datawrangling.com/
I had a post last year describing launching a cluster of the small 32 bit instances: http://www.datawrangling.com/mpi-cluster-with-python-and-amazon-ec2-part-2-of-3.html Since then, Amazon upgraded to 64 bit "extra-large" instances with full gigabit ethernet which might make this feasible. Essentially you are getting a full physical box, where with the small instances you would be sharing network, disk, etc. I built a set of new Fedora images and config scripts for the extra-large instances which include NFS,mpich,lam,openmpi,ganglia etc. I'd like to use a standard cluster distribution, but it would take some hacking to get the networking to work properly within EC2. Amazon uses a custom firewall setup where autodiscovery won't work, also multicast is not supported and subnets are randomly assigned. On Fri, Mar 7, 2008 at 1:18 PM, Peter St. John <[EMAIL PROTECTED]> wrote: > Peter Sk, > Where is the blog you mentioned (where you'll be posting followups)? > Thanks, > Peter St > > On Fri, Mar 7, 2008 at 12:49 PM, Peter Skomoroch < > [EMAIL PROTECTED]> wrote: > > > I'm running bonnie++ on a xlarge instance right now with 30 GB files on > > /mnt. I'll post the results when it finishes. I also have Ganglia set up > > on the node, so you can check that out until I shut the instance down: > > > > http://ec2-72-44-53-20.compute-1.amazonaws.com/ganglia > > > > > > On Fri, Mar 7, 2008 at 12:05 PM, Peter Skomoroch < > > [EMAIL PROTECTED]> wrote: > > > > > Joe, thanks for the feedback. The bonnie results were not actually > > > mine, I was just pointing to some numbers run by Paul Moen. > > > > > > Your 1GB file data is likely more representative, but with 15 GB ram, > > > > you need to be testing 30-60 GB files. > > > > > > > > > > I'll try to tweak the BPS bonnie tests to run some large files... > > > > > > > > > > > > On Fri, Mar 7, 2008 at 11:57 AM, Joe Landman < > > > [EMAIL PROTECTED]> wrote: > > > > > > > Peter Skomoroch wrote: > > > > > > > > > Extra Large Instance: > > > > > > > > > > 15 GB memory > > > > > 8 EC2 Compute Units (4 virtual cores with 2 EC2 Compute > > > > Units each) > > > > > 1,690 GB instance storage (4 x 420 GB plus 10 GB root > > > > partition) > > > > > 64-bit platform > > > > > I/O Performance: High > > > > > > > > Note: minor criticism, but overall, nice results. > > > > > > > > Looking over your bonnie results is worth a quick comment. Any time > > > > you > > > > have bonnie or IOzone (or other IO benchmarks) which are testing > > > > file > > > > sizes less than ram size, you are not actually measuring disk IO. > > > > This > > > > is cache speed pure and simple. Either page/buffer cache, or RAID > > > > cache, or whatever. > > > > > > > > We have had people tell us to our face that their 2GB file results > > > > (on a > > > > 16 GB RAM machine) were somehow indicative of real file performance, > > > > when, if they walked over to the units they were testing, they would > > > > have noticed the HD lights simply not blinking ... Yeah, an amusing > > > > beer story (the longer version of it), but a problem none-the-less. > > > > > > > > Your 1GB file data is likely more representative, but with 15 GB > > > > ram, > > > > you need to be testing 30-60 GB files. > > > > > > > > Not trying to be a marketing guy here or anything like that ... we > > > > test > > > > our JackRabbit units with 80GB to 1.3TB sized files. We see > > > > (sustained) > > > > 750 MB/s - 1.3 GB/s in these tests. We also note some serious > > > > issues > > > > with the linux buffer cache and multiple RAID controllers (buffer > > > > cache > > > > appears to serialize access). We do this as we actually want to > > > > measure > > > > disk performance, and not buffer cache performance. > > > > > > > > That criticism aside, nice results. It shows what a "cloud" can do. > > > > > > > > > Price: $0.80 per instance hour > > > > > > > > > > > > -- > > > > Joseph Landman, Ph.D > > > > Founder and CEO > > > > Scalable Informatics LLC, > > > > email: [EMAIL PROTECTED] > > > > web : http://www.scalableinformatics.com > > > > http://jackrabbit.scalableinformatics.com > > > > phone: +1 734 786 8423 > > > > fax : +1 866 888 3112 > > > > cell : +1 734 612 4615 > > > > > > > > > > > > > > > > -- > > > Peter N. Skomoroch > > > [EMAIL PROTECTED] > > > http://www.datawrangling.com > > > > > > > > > > > -- > > Peter N. Skomoroch > > [EMAIL PROTECTED] > > http://www.datawrangling.com > > > > _______________________________________________ > > Beowulf mailing list, [email protected] > > To change your subscription (digest mode or unsubscribe) visit > > http://www.beowulf.org/mailman/listinfo/beowulf > > > > > -- Peter N. Skomoroch [EMAIL PROTECTED] http://www.datawrangling.com
_______________________________________________ Beowulf mailing list, [email protected] To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
