for nehalem CPU and memory DIMM population setupplease checkout this link http://blogs.sun.com/jnerl/entry/configuring_and_optimizing_intel_xeon
that explain various memory bandwidth and DIMM population. Of course these memory bandwidth may not apply to lustre:-(
On 8/10/2010 7:49 AM, James Robnett wrote:
Hello List memebers, We are planning a Lustre setup for our lab. I was searching for cheap but reliable DAS and/or JBOD solutions for setting this up and came across this: http://www.supermicro.com/products/chassis/4U/846/SC846E26-R1200.cfm I would like to know if anyone has any experience in setting up this or similar kind of a system.I've only just started playing with 2 OSSes based on that chassis. They have a: Supermicro X8DTH (7 PCI-E 8x slots) motherboard Dual quad core Intel Xeon processors Four 8 port 3ware 9650 raid controllers (4 OSTs/OSS) 1 Mellanox MT26428 QDR Infiniband HCA 24 WD2003FYYS hard drives The disks are carved into four 4+1+spare R5 arrays for 8 total OST's. I hate to admit it but I skimped on the memory initially and only got 4GB. We'll never have a read cache hit regardless so I'm not entirely convinced it's starved at 4GB but the manual suggests that's low for 4 OST's and 2 processors (though empirically I'm not convinced). If you can throw enough memory at it to get disk cache hits by all means do. I'll throw more memory at it at the first hint it's beneficial. I'm currently client bound doing total Lustre throughput tests. For reasons I don't fully understand (and have been pondering posting about) I simply can't get more than about 700MB/s reads on a client. They seem to be bound re-assembling the replies which I expected, I just assumed the peak would be higher. With 3 clients I get 2.1GB/s reads across the 8 OST's with IOzone: 8 threads per 3 discrete IOzones, doing 1MB IO's so each OST is seeing 3 concurrent reads which more or less mimics our software. Eventually I expect it to peak at around 3GB/s aggregate (1.5GB/s per OSS). Our data reduction software/cluster size lends itself to this type of config where we have many times the number of OST's of multi-GB files mapped to individual cluster nodes for processing. So no striping; each file on an OST. The disks are relatively reliable. I don't plan to scale it beyond 6-8 OSS's so reliability is still manageable. We use the same chassis/disks but different raid/network link for bulk storage. We have 22 in all (~500 disks). James Robnett NRAO/AOC _______________________________________________ Lustre-discuss mailing list [email protected] http://lists.lustre.org/mailman/listinfo/lustre-discuss
<<attachment: laotsao.vcf>>
_______________________________________________ Lustre-discuss mailing list [email protected] http://lists.lustre.org/mailman/listinfo/lustre-discuss
