Hello, I've got a strange problem with lustre read perforamnce.
Following setup: 8 Core (100GB RAM) combined MGS/OSS with two OST targets server, with Infiniband 2x RAID6 (11x 1 TB HDD -> 8TB usable on each RAID). (/dev/sdc and /dev/sdd) /dev/sdc1 = 1024GB /dev/sdd1 = 1024GB 1x small HDD for journal and MDS. (/dev/sdb) /dev/sdb1 = 10 GB journal /dev/sdb2 = 10GB journal /dev/sdb3 = 50 GB MDS Procedure: mkfs.lustre --reformat --fsname lustre --mdt --mgs /dev/sdb3 mkfs.lustre --fsname=lustre --reformat --ost --mgsnode 192.168....@o2ib --mkfsoptions="-E stride=256 -E stripe-width=9 -J device=/dev/sdb1" /dev/sdc1 mkfs.lustre --fsname=lustre --reformat --ost --mgsnode 192.168....@o2ib --mkfsoptions="-E stride=256 -E stripe-width=9 -J device=/dev/sdb2" /dev/sdd1 Client machine: 24 core (50 GB RAM) with infiniband. Mount : mount -t lustre 192.168....@o2ib:/lustre /mnt/client/<mailto:192.168....@o2ib:/lustre%20/mnt/client/> After that I ran some simple throughput tests. Write performance (5 threads) is about 1072 MB/s (which is great), but read performance is very bad ~ 510 MB/s. What could be wrong about this setup? Why write performance is so good, but read so bad? Of course I tested the hardware itself, made a filesystem on the raids (without lustre) and ran the same tests, ~1,3GB/s write 1,7GB/s read (800 GB file) The infiniband is also okay, used the same cards/cable for some HPC tests just hours before, everything good. Nevertheless used the server as a client two (just to bypass the infiniband side as a potential failure). Nearly the same results (write good, read bad). It's like there is a wall at ~510 MB/s read. Any ideas or suggestions ? Its really urgent. Thank you a lot! 2.6.27.39+lustre1.8.3+0.credativ.lenny.4 Lustre version: 1.8.3-20100410022943-PRISTINE-2.6.27.39+lustre1.8.3+0.credativ.lenny.4 sysGen GmbH Sergius Siczek Am Hallacker 48 · 28327 Bremen Tel (0421) 4 09 66-32 · Fax (0421) 4 09 66-33 [email protected]<mailto:[email protected]> · www.sysgen.de<http://www.sysgen.de/> Geschäftsführung: Gabriele Nikisch, Anke Heinßen · Eingetragen beim Amtsgericht Walsrode · HRB 121943
_______________________________________________ Lustre-discuss mailing list [email protected] http://lists.lustre.org/mailman/listinfo/lustre-discuss
