Re: [zfs-discuss] COMSTAR iSCSI and two Windows computers
NTFS is not a clustered file system and thus can't handle multiple clients accessing the data. You could use MelioFS if you're Windows based, which handles metadata updates and locking between the accessing nodes to be able to share a NTFS disk between them - even over iSCSI. MelioFS: http://www.sanbolic.com/melioFS.htm -Arve Hi guys I wanted to ask how i could setup a iSCSI device to be shared by 2 computers concurrently, by that i mean sharing files like it was a NFS share but use iSCSI instead. I tried and setup iSCSI on both computers and was able to see my files (I had formatted it NTFS before), from my laptop I uploaded a 400MB video file to the root directory and from my desktop I browsed the same directory and the file was not there?? Thanks -- ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] COMSTAR iSCSI and two Windows computers
And.. if you're using iSCSI towards ZFS and want to have shared access, take a look at GlusterFS which you use in front of multiple ZFS nodes as accessing point. GlusterFS: http://www.gluster.org/ -Arve -Original Message- From: zfs-discuss-boun...@opensolaris.org [mailto:zfs-discuss- boun...@opensolaris.org] On Behalf Of Giovanni Sent: 18. juni 2010 07:44 To: zfs-discuss@opensolaris.org Subject: [zfs-discuss] COMSTAR iSCSI and two Windows computers Hi guys I wanted to ask how i could setup a iSCSI device to be shared by 2 computers concurrently, by that i mean sharing files like it was a NFS share but use iSCSI instead. I tried and setup iSCSI on both computers and was able to see my files (I had formatted it NTFS before), from my laptop I uploaded a 400MB video file to the root directory and from my desktop I browsed the same directory and the file was not there?? Thanks -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] SSDs adequate ZIL devices?
Not to forget the The Deneva Reliability disks from OCZ that just got released. See http://www.oczenterprise.com/details/ocz-deneva-reliability-2-5-emlc-ssd.htm l The Deneva Reliability family features built-in supercapacitor (SF-1500 models) that acts as a temporary power backup in the event of sudden power loss, and enables the drive to complete its task ensuring no data loss. -Arve -Original Message- From: zfs-discuss-boun...@opensolaris.org [mailto:zfs-discuss- boun...@opensolaris.org] On Behalf Of Christopher George Sent: 16. juni 2010 00:47 To: zfs-discuss@opensolaris.org Subject: Re: [zfs-discuss] SSDs adequate ZIL devices? So why buy SSD for ZIL at all? For the record, not all SSDs ignore cache flushes. There are at least two SSDs sold today that guarantee synchronous write semantics; the Sun/Oracle LogZilla and the DDRdrive X1. Also, I believe it is more accurate to describe the root cause as not power protecting on-board volatile caches. As the X25-E does implement the ATA FLUSH CACHE command, but does not have the required power protection to avoid transaction (data) loss. Best regards, Christopher George Founder/CTO www.ddrdrive.com -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] OCZ Devena line of enterprise SSD
Got prices from a retailer now: 100GB - DENRSTE251E10-0100~1100 USD 200GB - DENRSTE251E10-0200~1900 USD 400GB - DENRSTE251E10-0400~4500 USD Prices were given to a country in Europe, so USD prices might be lower. -Arve -Original Message- From: zfs-discuss-boun...@opensolaris.org [mailto:zfs-discuss- boun...@opensolaris.org] On Behalf Of Scott Meilicke Sent: 15. juni 2010 22:10 To: zfs-discuss@opensolaris.org Subject: Re: [zfs-discuss] OCZ Devena line of enterprise SSD Price? I cannot find it. -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] OCZ Devena line of enterprise SSD
They'll probably track me down and shoot me later on. :o -A -Original Message- From: David Magda [mailto:dma...@ee.ryerson.ca] Sent: 16. juni 2010 15:03 To: Arve Paalsrud Cc: 'Scott Meilicke'; zfs-discuss@opensolaris.org Subject: Re: [zfs-discuss] OCZ Devena line of enterprise SSD On Wed, June 16, 2010 07:59, Arve Paalsrud wrote: Got prices from a retailer now: 100GB - DENRSTE251E10-0100~1100 USD 200GB - DENRSTE251E10-0200~1900 USD 400GB - DENRSTE251E10-0400~4500 USD Prices were given to a country in Europe, so USD prices might be lower. Heh. I just did a quick search on those model numbers, and three hits appear: 1. OCZ's web page 2. the Jive interface to zfs-discuss 3. the mailman archive of zfs-discuss I think OCZ's marketing department needs to work a little harder. :) ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] High-Performance ZFS (2000MB/s+)
Hi, We are currently building a storage box based on OpenSolaris/Nexenta using ZFS. Our hardware specifications are as follows: Quad AMD G34 12-core 2.3 GHz (~110 GHz) 10 Crucial RealSSD (6Gb/s) 42 WD RAID Ed. 4 2TB disks + 6Gb/s SAS expanders LSI2008SAS (two 4x ports) Mellanox InfiniBand 40 Gbit NICs 128 GB RAM This setup gives us about 40TB storage after mirror (two disks in spare), 2.5TB L2ARC and 64GB Zil, all fit into a single 5U box. Both L2ARC and Zil shares the same disks (striped) due to bandwidth requirements. Each SSD has a theoretical performance of 40-50k IOPS on 4k read/write scenario with 70/30 distribution. Now, I know that you should have mirrored Zil for safety, but the entire box are synchronized with an active standby on a different site location (18km distance - round trip of 0.16ms + equipment latency). So in case the Zil in Site A takes a fall, or the motherboard/disk group/motherboard dies - we still have safety. DDT requirements for dedupe on 16k blocks should be about 640GB when main pool are full (capacity). Without going into details about chipsets and such, do any of you on this list have any experience with a similar setup and can share with us your thoughts, do's and dont's, and any other information that could be of help while building and configuring this? What I want to achieve is 2 GB/s+ NFS traffic against our ESX clusters (also InfiniBand-based), with both dedupe and compression enabled in ZFS. Let's talk moon landings. Regards, Arve -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] High-Performance ZFS (2000MB/s+)
On Tue, Jun 15, 2010 at 3:09 PM, Erik Trimble erik.trim...@oracle.comwrote: On 6/15/2010 4:42 AM, Arve Paalsrud wrote: Hi, We are currently building a storage box based on OpenSolaris/Nexenta using ZFS. Our hardware specifications are as follows: Quad AMD G34 12-core 2.3 GHz (~110 GHz) 10 Crucial RealSSD (6Gb/s) 42 WD RAID Ed. 4 2TB disks + 6Gb/s SAS expanders LSI2008SAS (two 4x ports) Mellanox InfiniBand 40 Gbit NICs 128 GB RAM This setup gives us about 40TB storage after mirror (two disks in spare), 2.5TB L2ARC and 64GB Zil, all fit into a single 5U box. Both L2ARC and Zil shares the same disks (striped) due to bandwidth requirements. Each SSD has a theoretical performance of 40-50k IOPS on 4k read/write scenario with 70/30 distribution. Now, I know that you should have mirrored Zil for safety, but the entire box are synchronized with an active standby on a different site location (18km distance - round trip of 0.16ms + equipment latency). So in case the Zil in Site A takes a fall, or the motherboard/disk group/motherboard dies - we still have safety. DDT requirements for dedupe on 16k blocks should be about 640GB when main pool are full (capacity). Without going into details about chipsets and such, do any of you on this list have any experience with a similar setup and can share with us your thoughts, do's and dont's, and any other information that could be of help while building and configuring this? What I want to achieve is 2 GB/s+ NFS traffic against our ESX clusters (also InfiniBand-based), with both dedupe and compression enabled in ZFS. Let's talk moon landings. Regards, Arve Given that for ZIL, random write IOPS is paramount, the RealSSD isn't a good choice. SLC SSDs still spank any MLC device, and random IOPS for something like an Intel X25-E or OCZ Vertex EX are over twice that of the RealSSD. I don't know where they manage to get 40k+ IOPS number for the RealSSD (I know it's in the specs, but how did they get that?), but that's not what others are reporting: http://benchmarkreviews.com/index.php?option=com_contenttask=viewid=454Itemid=60limit=1limitstart=7 See http://www.anandtech.com/show/2944/3 and http://www.crucial.com/pdf/Datasheets-letter_C300_RealSSD_v2-5-10_online.pdf But I agree that we should look into using the Vertex instead. Sadly, none of the current crop of SSDs support a capacitor or battery to back up their local (on-SSD) cache, so they're all subject to data loss on a power interruption. Noted Likewise, random Read dominates L2ARC usage. Here, the most cost-effective solutions tend to be MLC-based SSDs with more moderate IOPS performance - the Intel X25-M and OCZ Vertex series are likely much more cost-effective than a RealSSD, especially considering price/performance. Our other option are to use two Fusion-IO ioDrive Duo SLC/MLC or the SMLC when available (as well as drivers for Solaris) - so the price we're currently talking about is not an issue. Also, given the limitations of a x4 port connection to the rest of the system, I'd consider using a couple more SAS controllers, and fewer Expanders. The SSDs together are likely to be able to overwhelm a x4 PCI-E connection, so I'd want at least one dedicated x4 SAS HBA just for them. For the 42 disks, it depends more on what your workload looks like. If it is mostly small or random I/O to the disks, you can get away with fewer HBAs. Large, sequential I/O to the disks is going to require more HBAs. Remember, a modern 7200RPM SATA drive can pump out well over 100MB/s sequential, but well under 10MB/s random. Do the math to see how fast it will overwhelm the x4 PCI-E 2.0 connection which maxes out at about 2GB/s. We're talking about 4X SAS 6Gb/s lanes - 4800MB/s per port. See http://www.lsi.com/DistributionSystem/AssetDocument/SCG_LSISAS2008_PB_043009.pdffor specifications of the LSI chip. In other words, it utilizes PCIe 2.0 8x. I'd go with 2 Intel X25-E 32GB models for ZIL. Mirror them - striping isn't really going to buy you much here (so far as I can tell). 6Gbit/s SAS is wasted on HDs, so don't bother paying for it if you can avoid doing so. Really, I'd suspect that paying for 6Gb/s SAS isn't worth it at all, as really only the read performance of the L2ARC SSDs might possibly exceed 3Gb/s SAS. What about bandwidth in this scenario? Won't the ZIL be limited to the throughput of only one X25-E? The SATA disks operates at 3Gb/s through the SAS expanders, so no 6Gb/s there. I'm going to say something sacrilegious here: 128GB of RAM may be overkill. You have the SSDs for L2ARC - much of which will be the DDT, but, if I'm reading this correctly, even if you switch to the 160GB Intel X25-M, that give you 8 x 160GB = 1280GB of L2ARC, of which only half is in-use by the DDT. The rest is file cache. You'll need lots of RAM if you plan on storing lots of small files in the L2ARC (that is, if your workload is lots of small
Re: [zfs-discuss] High-Performance ZFS (2000MB/s+)
On Tue, Jun 15, 2010 at 4:20 PM, Erik Trimble erik.trim...@oracle.comwrote: On 6/15/2010 6:57 AM, Arve Paalsrud wrote: On Tue, Jun 15, 2010 at 3:09 PM, Erik Trimble erik.trim...@oracle.comwrote: I'd go with 2 Intel X25-E 32GB models for ZIL. Mirror them - striping isn't really going to buy you much here (so far as I can tell). 6Gbit/s SAS is wasted on HDs, so don't bother paying for it if you can avoid doing so. Really, I'd suspect that paying for 6Gb/s SAS isn't worth it at all, as really only the read performance of the L2ARC SSDs might possibly exceed 3Gb/s SAS. What about bandwidth in this scenario? Won't the ZIL be limited to the throughput of only one X25-E? The SATA disks operates at 3Gb/s through the SAS expanders, so no 6Gb/s there. Yes - though I'm not sure how the slog devices work when there is more than one. I *don't* think they work like the L2ARC devices, which work round-robin. You'd have to ask. If they're doing a true stripe, then I doubt you'll get much more performance as weird as that sounds. Also, even with a single X25-E, you can service a huge number of IOPS - likely more small IOPS than can be pushed over even an Infiniband interface. The place that the Infiniband would certainly outpace the X25-E's capacity is for large writes, where a single 100MB write would suck up all the X25-E's throughput capability. But the Intel X25-E are limited to about 200 MB/s write, regardless of IOPS. So when throwing a lot of 16k IOPS (about 13 000) at it, it will still be limited to 200 MB/s - or about 6-7% of the throughput of an QDR InfiniBand links capacity. So I hereby officially ask: Can I have multiple slogs striped to handle higher bandwidth than a single device can - is that supported in ZFS? -- Erik Trimble Java System Support Mailstop: usca22-123 Phone: x17195 Santa Clara, CA - Arve ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] High-Performance ZFS (2000MB/s+)
On Tue, Jun 15, 2010 at 3:33 PM, Erik Trimble erik.trim...@oracle.comwrote: On 6/15/2010 6:17 AM, Darren J Moffat wrote: On 15/06/2010 14:09, Erik Trimble wrote: I'm going to say something sacrilegious here: 128GB of RAM may be overkill. You have the SSDs for L2ARC - much of which will be the DDT, The point of L2ARC is that you start adding L2ARC when you can no longer physically put in (or afford) to add any more DRAM, so if OP can afford to put in 128GB of RAM then they should. True. I was speaking price/performance. Those 8GB DIMMs are still pretty darned pricey... -- Erik Trimble Java System Support Mailstop: usca22-123 Phone: x17195 Santa Clara, CA The motherboard has 32 DIMM slots - making use of 32 4GB modules to gain 128GB quite affordable :) -Arve ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] High-Performance ZFS (2000MB/s+)
-Original Message- From: Garrett D'Amore [mailto:garr...@nexenta.com] Sent: 15. juni 2010 17:43 To: Arve Paalsrud Cc: zfs-discuss@opensolaris.org Subject: Re: [zfs-discuss] High-Performance ZFS (2000MB/s+) On Tue, 2010-06-15 at 04:42 -0700, Arve Paalsrud wrote: Hi, We are currently building a storage box based on OpenSolaris/Nexenta using ZFS. Our hardware specifications are as follows: Quad AMD G34 12-core 2.3 GHz (~110 GHz) 10 Crucial RealSSD (6Gb/s) 42 WD RAID Ed. 4 2TB disks + 6Gb/s SAS expanders LSI2008SAS (two 4x ports) Mellanox InfiniBand 40 Gbit NICs Just recognize that those NICs are IB only. Solaris currently does not support 10GbE using Mellanox products, even though other operating systems do. (There are folks working on resolving this, but I think we're still a couple months from seeing the results of that effort.) 128 GB RAM This setup gives us about 40TB storage after mirror (two disks in spare), 2.5TB L2ARC and 64GB Zil, all fit into a single 5U box. Both L2ARC and Zil shares the same disks (striped) due to bandwidth requirements. Each SSD has a theoretical performance of 40-50k IOPS on 4k read/write scenario with 70/30 distribution. Now, I know that you should have mirrored Zil for safety, but the entire box are synchronized with an active standby on a different site location (18km distance - round trip of 0.16ms + equipment latency). So in case the Zil in Site A takes a fall, or the motherboard/disk group/motherboard dies - we still have safety. I expect that you need more space for L2ARC and a lot less for Zil. Furthmore, you'd be better served by an even lower latency/higher IOPs ZIL. If you're going to spend this kind of cash, I think I'd recommend at least one or two DDR Drive X1 units or something similar. While not very big, you don't need much to get a huge benefit from the ZIL, and I think the vastly superior IOPS of these units will pay off in the end. What about the ZIL bandwidth in this case? I mean, could I stripe across multiple devices to be able to handle higher throughput? Otherwise I would still be limited to the performance of the unit itself (155 MB/s). DDT requirements for dedupe on 16k blocks should be about 640GB when main pool are full (capacity). Dedup is not always a win, I think. I'd look hard at your data and usage to determine whether to use it. -- Garrett -Arve ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
