On 15/06/2010 12:42, "Arve Paalsrud" <arve.paals...@gmail.com> wrote:
> Hi, > > We are currently building a storage box based on OpenSolaris/Nexenta using > ZFS. > Our hardware specifications are as follows: > > Quad AMD G34 12-core 2.3 GHz (~110 GHz) > 10 Crucial RealSSD (6Gb/s) > 42 WD RAID Ed. 4 2TB disks + 6Gb/s SAS expanders > LSI2008SAS (two 4x ports) > Mellanox InfiniBand 40 Gbit NICs I was told that IB support in Nexenta is scheduled to be released in 3.0.4 (beginning of July). > 128 GB RAM > > This setup gives us about 40TB storage after mirror (two disks in spare), > 2.5TB L2ARC and 64GB Zil, all fit into a single 5U box. > > Both L2ARC and Zil shares the same disks (striped) due to bandwidth > requirements. Each SSD has a theoretical performance of 40-50k IOPS on 4k > read/write scenario with 70/30 distribution. Now, I know that you should have > mirrored Zil for safety, but the entire box are synchronized with an active > standby on a different site location (18km distance - round trip of 0.16ms + > equipment latency). So in case the Zil in Site A takes a fall, or the > motherboard/disk group/motherboard dies - we still have safety. > > DDT requirements for dedupe on 16k blocks should be about 640GB when main pool > are full (capacity). > > Without going into details about chipsets and such, do any of you on this list > have any experience with a similar setup and can share with us your thoughts, > do's and dont's, and any other information that could be of help while > building and configuring this? > > What I want to achieve is 2 GB/s+ NFS traffic against our ESX clusters (also > InfiniBand-based), with both dedupe and compression enabled in ZFS. As VMware does not currently support NFS over RDMA, you will need to stick with IPoIB which will suffer from some performance implications inherent to traditional TCP/IP stack. You could also use iSER or SRP which are both supported. > > Let's talk moon landings. > > Regards, > Arve -- Przem _______________________________________________ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
