Re: [zfs-discuss] JBOD recommendation for ZFS usage
So, if I may, is this the correct summary of the answer to original question (on JBOD for a ZFS HA cluster): === SC847E26-RJBOD1 with dual-ported SAS drives are known to work in a failover HA storage scenario, allowing both servers (HBAs) access to each single SAS drive individually, so zpools can be configured from any disks regardless of which backplane they are connected to. HA Clusterware, such as NexentaStor HA-Cluster plugin should be used to ensure that only one head node actually uses a given disk drive in an imported ZFS pool. === Is the indented statement correct? :) Other questions: What clusterware would be encouraged now for OpenIndiana boxes? Also, in case of clustered shared filesystems (like VMWare vmfs) can these JBODs allow two different servers to access one drive simultaneously in a safe manner (do they do fencing, reservations and other SCSI magic)? Following up on some of this forum's discussions, I read the manuals on SuperMicro's SC847E26-RJBOD1 this weekend. We see quite a few of these in the NexentaStor installed base. The NexentaStor HA-Cluster plugin manages STONITH and reservations. I do not believe programming expanders or switches for clustering is the best approach. It is better to let the higher layers manage this. The cost of a SATA disk + SATA/SAS interposer is about the same as a native SAS drive. Native SAS makes a better solution. -- ++ || | Климов Евгений, Jim Klimov | | технический директор CTO | | ЗАО ЦОС и ВТ JSC COSHT | || | +7-903-7705859 (cellular) mailto:jimkli...@cos.ru | |CC:ad...@cos.ru,jimkli...@gmail.com | ++ | () ascii ribbon campaign - against html mail | | /\- against microsoft attachments | ++ ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] JBOD recommendation for ZFS usage
Thomas, You can consider the DataON DNS-1600(4U 24 3.5 Bay 6Gb/s SAS JBOD). It is perfect for ZFS storage as the alternative of J4400. http://dataonstorage.com/dns-1600 And we recommend you to use native SAS HD like Seagate Constellation ES 2TB SAS to connect 2 hosts for fail-over cluster. The following is setup diagram of HA failover cluster with Nexenta. Same configuration can applied to Solaris, OpenSolaris and OpenIndiana http://dataonstorage.com/nexentaha We also have DSM(Disk Shelf Management Tool) available for Solaris 10 and Nexenta to help identify fail disk and JBOD. You can also check the status of all FRU http://dataonstorage.com/dsm FYI, we have reseller in Germany. If you need the additional info, you can let me know! Rocky -Original Message- From: zfs-discuss-boun...@opensolaris.org [mailto:zfs-discuss-boun...@opensolaris.org] On Behalf Of Thomas Nau Sent: Sunday, May 29, 2011 11:07 PM To: zfs-discuss@opensolaris.org Subject: [zfs-discuss] JBOD recommendation for ZFS usage Dear all Sorry if it's kind of off-topic for the list but after talking to lots of vendors I'm running out of ideas... We are looking for JBOD systems which (1) hold 20+ 3.3 SATA drives (2) are rack mountable (3) have all the nive hot-swap stuff (4) allow 2 hosts to connect via SAS (4+ lines per host) and see all available drives as disks, no RAID volume. In a perfect world both hosts would connect each using two independent SAS connectors The box will be used in a ZFS Solaris/based fileserver in a fail-over cluster setup. Only one host will access a drive at any given time. It seems that a lot of vendors offer JBODs but so far I haven't found one in Germany which handles (4). Any hints? ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] JBOD recommendation for ZFS usage
Dear all Sorry if it's kind of off-topic for the list but after talking to lots of vendors I'm running out of ideas... We are looking for JBOD systems which (1) hold 20+ 3.3 SATA drives (2) are rack mountable (3) have all the nive hot-swap stuff (4) allow 2 hosts to connect via SAS (4+ lines per host) and see all available drives as disks, no RAID volume. In a perfect world both hosts would connect each using two independent SAS connectors The box will be used in a ZFS Solaris/based fileserver in a fail-over cluster setup. Only one host will access a drive at any given time. It seems that a lot of vendors offer JBODs but so far I haven't found one in Germany which handles (4). Any hints? I know of [1] and [2]. The Promise one's are available for purchase through retail channels; don't know about the Xyratex. Neither do I know about quality, but AFAIK Xyratex is or was main supplier of NetApp's JBOD, which I know to be of good quality. Regards Florian Wagner [1] http://promise.com/storage/raid_series.aspx?region=en-USm=577sub_m=sub_m_2rsn1=1rsn3=25 [2] http://www.xyratex.com/products/storage-systems/onestor-SP2424s.aspx signature.asc Description: PGP signature ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] JBOD recommendation for ZFS usage
Following up on some of this forum's discussions, I read the manuals on SuperMicro's SC847E26-RJBOD1 this weekend. At the very least, this box provides dual-expander backplanes (2 BPs for a total of 45 hot-swap disks), so each JBOD has 4 outgoing SFF8087 (4xSATA iPass) connectors. However it seems that the backplane chips are doubled for multipathing-failover within a single head node with dual connections from same or different HBAs. (Further on, backplanes may be daisy-chained to attach other JBODs to the same HBA path - but if you don't care much for bandwidth limitations). According to the docs, each chip addresses all disks on its backplane, and it seems implied (but not expressly stated) that either one chip and path works, or another. So if your application can live with the unit of failover being a bunch of 21 or 24 disks - that might be a way to go. However each head would only have one connection to each backplane, and I'm not sure if you can STONITH the non-leading head to enforce failovers (and enable the specific PRI/SEC chip of the backplane). Also one point was stressed many times in the docs: these failover backplanes require use of SAS drives, no SATA (while the single-path BPs are okay with both SAS and SATA). Still, according to the forums, SATA disks on shared backplanes often give too much headache and may give too little performance in comparison... I am not sure if this requirement also implies dual SAS data connectors - pictures of HCL HDDs all have one connector... Still, I gess my post poses mre questions than answers, but maybe some other list readers can reply... Hint: Nexenta people seem to be good OEM friends with Supermicro, so they might know ;) HTH, //Jim Klimov ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] JBOD recommendation for ZFS usage
Thanks Jim and all the other who have replied so far On 05/30/2011 11:37 AM, Jim Klimov wrote: ... So if your application can live with the unit of failover being a bunch of 21 or 24 disks - that might be a way to go. However each head would only have one connection to each backplane, and I'm not sure if you can STONITH the non-leading head to enforce failovers (and enable the specific PRI/SEC chip of the backplane). That exactly my point. I don't need any internal failover which restricts which disks a host can see. We want to failover between hosts, not connections. For the later we would use another JBOD and let ZFS do the dirty job or mirroring. We run a similar setup for years but with FC connected RAID systems. Over time they are kind of limited when it comes to price/performance Also one point was stressed many times in the docs: these failover backplanes require use of SAS drives, no SATA (while the single-path BPs are okay with both SAS and SATA). Still, according to the forums, SATA disks on shared backplanes often give too much headache and may give too little performance in comparison... I would be fine with SAS as well Thomas ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] JBOD recommendation for ZFS usage
On Mon, May 30, 2011 at 08:06:31AM +0200, Thomas Nau wrote: We are looking for JBOD systems which (1) hold 20+ 3.3 SATA drives (2) are rack mountable (3) have all the nive hot-swap stuff (4) allow 2 hosts to connect via SAS (4+ lines per host) and see all available drives as disks, no RAID volume. In a perfect world both hosts would connect each using two independent SAS connectors The box will be used in a ZFS Solaris/based fileserver in a fail-over cluster setup. Only one host will access a drive at any given time. I'm using a J4200 array as shared storage for a cluster. It needs a SAS HBA in each cluster node. The disks in the array are visible to both nodes in the cluster. Here's the feature list. I don't know if it's still available: Sun Storage J4200 Array: # Scales up to 48 SAS/SATA disk drives # Provides up to 72 Gb/sec of total bandwidth * Up to 72 Gb/sec of total bandwidth * Four x4-wide 3 Gb/sec SAS host/uplink ports (48 Gb/sec bandwidth) * Two x4-wide 3 Gb/sec SAS expansion ports (24 Gb/sec bandwidth) * Scales up to 48 drives -- -Gary Mills--Unix Group--Computer and Network Services- ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] JBOD recommendation for ZFS usage
On May 30, 2011, at 2:37 AM, Jim Klimov wrote: Following up on some of this forum's discussions, I read the manuals on SuperMicro's SC847E26-RJBOD1 this weekend. We see quite a few of these in the NexentaStor installed base. The other commonly found 3.5 24-drive JBOD is the DataON DNS-1600, a product staggeringly similar to Sun's J4400. At the very least, this box provides dual-expander backplanes (2 BPs for a total of 45 hot-swap disks), so each JBOD has 4 outgoing SFF8087 (4xSATA iPass) connectors. However it seems that the backplane chips are doubled for multipathing-failover within a single head node with dual connections from same or different HBAs. (Further on, backplanes may be daisy-chained to attach other JBODs to the same HBA path - but if you don't care much for bandwidth limitations). We also commonly see the dual-expander backplanes. According to the docs, each chip addresses all disks on its backplane, and it seems implied (but not expressly stated) that either one chip and path works, or another. For SAS targets, both paths work simultaneously. So if your application can live with the unit of failover being a bunch of 21 or 24 disks - that might be a way to go. However each head would only have one connection to each backplane, and I'm not sure if you can STONITH the non-leading head to enforce failovers (and enable the specific PRI/SEC chip of the backplane). The NexentaStor HA-Cluster plugin manages STONITH and reservations. I do not believe programming expanders or switches for clustering is the best approach. It is better to let the higher layers manage this. Also one point was stressed many times in the docs: these failover backplanes require use of SAS drives, no SATA (while the single-path BPs are okay with both SAS and SATA). Still, according to the forums, SATA disks on shared backplanes often give too much headache and may give too little performance in comparison... The cost of a SATA disk + SATA/SAS interposer is about the same as a native SAS drive. Native SAS makes a better solution. I am not sure if this requirement also implies dual SAS data connectors - pictures of HCL HDDs all have one connector... These are dual ported. Still, I gess my post poses mre questions than answers, but maybe some other list readers can reply... Hint: Nexenta people seem to be good OEM friends with Supermicro, so they might know ;) Yes :-) -- richard ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] JBOD recommendation for ZFS usage
Thanks, now I have someone to interrogate, who seems to have seen these boxes live - if you don't mind ;) - Original Message - From: Richard Elling richard.ell...@gmail.com Date: Monday, May 30, 2011 22:04 We also commonly see the dual-expander backplanes. According to the docs, each chip addresses all disks on its backplane, and it seems implied (but not expressly stated) that either one chip and path works, or another. For SAS targets, both paths work simultaneously. Does this mean that if the J0 uplinks of backplanes are connected to HBAs in two different servers, both of these servers can address individual disks (and the unit of failover is not a backplane but a disk after all)? And if both HBAs are in a single server, this doubles the SAS link throughput by having two paths - and can ZFS somehow balance among them? So if your application can live with the unit of failover being a bunch of 21 or 24 disks - that might be a way to go. However each head would only have one connection to each backplane, and I'm not sure if you can STONITH the non- leading head to enforce failovers (and enable the specific PRI/SEC chip of the backplane). The NexentaStor HA-Cluster plugin manages STONITH and reservations. I do not believe programming expanders or switches for clustering is the best approach. It is better to let the higher layers manage this. Makes sense. Since I originally thought that only one path works at a given time, it may be needed to somehow shutdown the competitor HBA/link ;) I am not sure if this requirement also implies dual SAS data connectors - pictures of HCL HDDs all have one connector... These are dual ported. Does this mean mecanically two 7-pin SATA data ports and a wide power port, for a total of 3 connectors on the back of HDD, as well as on the backplane sockets? Or does it mean something else? Because I've looked up half a dozen of SuperMicro-supported drives (bold SAS in the list for E2-series chassis), and in the online shops' images they all have the standard 2 connectors (wide and 7-pin): http://www.supermicro.com.tr/SAS-1-CompList.pdf The HCL is rather small, and other components may work but are not supported by SuperMicro. And to be more specific, do you know if Hitachi 7K3000 series SAS models HUS723020ALS640 (2Tb) or HUS723030ALS640 (3Tb) are suitable for these boxes? Does it make sense to keep the OS/swap on faster smaller drives like a mirror of HUS156030VLS600 (300Gb SAS 15kRPM) - or is it a waste of money? (And are they known to work in these boxes?) Hint: Nexenta people seem to be good OEM friends with Supermicro, so they might know ;) Yes :-) -- richard Thanks! //Jim Klimov ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] JBOD recommendation for ZFS usage
On Mon, May 30, 2011 at 1:35 PM, Jim Klimov j...@cos.ru wrote: Thanks, now I have someone to interrogate, who seems to have seen these boxes live - if you don't mind ;) - Original Message - From: Richard Elling richard.ell...@gmail.com Date: Monday, May 30, 2011 22:04 We also commonly see the dual-expander backplanes. According to the docs, each chip addresses all disks on its backplane, and it seems implied (but not expressly stated) that either one chip and path works, or another. For SAS targets, both paths work simultaneously. Does this mean that if the J0 uplinks of backplanes are connected to HBAs in two different servers, both of these servers can address individual disks (and the unit of failover is not a backplane but a disk after all)? And if both HBAs are in a single server, this doubles the SAS link throughput by having two paths - and can ZFS somehow balance among them? So if your application can live with the unit of failover being a bunch of 21 or 24 disks - that might be a way to go. However each head would only have one connection to each backplane, and I'm not sure if you can STONITH the non- leading head to enforce failovers (and enable the specific PRI/SEC chip of the backplane). The NexentaStor HA-Cluster plugin manages STONITH and reservations. I do not believe programming expanders or switches for clustering is the best approach. It is better to let the higher layers manage this. Makes sense. Since I originally thought that only one path works at a given time, it may be needed to somehow shutdown the competitor HBA/link ;) I am not sure if this requirement also implies dual SAS data connectors - pictures of HCL HDDs all have one connector... These are dual ported. Does this mean mecanically two 7-pin SATA data ports and a wide power port, for a total of 3 connectors on the back of HDD, as well as on the backplane sockets? Or does it mean something else? Because I've looked up half a dozen of SuperMicro-supported drives (bold SAS in the list for E2-series chassis), and in the online shops' images they all have the standard 2 connectors (wide and 7-pin): http://www.supermicro.com.tr/SAS-1-CompList.pdf The HCL is rather small, and other components may work but are not supported by SuperMicro. And to be more specific, do you know if Hitachi 7K3000 series SAS models HUS723020ALS640 (2Tb) or HUS723030ALS640 (3Tb) are suitable for these boxes? Does it make sense to keep the OS/swap on faster smaller drives like a mirror of HUS156030VLS600 (300Gb SAS 15kRPM) - or is it a waste of money? (And are they known to work in these boxes?) Hint: Nexenta people seem to be good OEM friends with Supermicro, so they might know ;) Yes :-) -- richard Thanks! //Jim Klimov SAS drives are SAS drives, they aren't like SCSI. There aren't 20 different versions with different pinouts. Multipathing is handled by mpxio. --Tim ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] JBOD recommendation for ZFS usage
Dunno about Germany, but LSI and DataON both have offerings. (The LSI units are probably going fast, as LSI exits that business having sold that unit to NetApp.) -- Garrett D'Amore On May 30, 2011, at 10:08 AM, Thomas Nau thomas@uni-ulm.de wrote: Dear all Sorry if it's kind of off-topic for the list but after talking to lots of vendors I'm running out of ideas... We are looking for JBOD systems which (1) hold 20+ 3.3 SATA drives (2) are rack mountable (3) have all the nive hot-swap stuff (4) allow 2 hosts to connect via SAS (4+ lines per host) and see all available drives as disks, no RAID volume. In a perfect world both hosts would connect each using two independent SAS connectors The box will be used in a ZFS Solaris/based fileserver in a fail-over cluster setup. Only one host will access a drive at any given time. It seems that a lot of vendors offer JBODs but so far I haven't found one in Germany which handles (4). Any hints? ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] JBOD recommendation for ZFS usage
Tim Cook wrote: SAS drives are SAS drives, they aren't like SCSI. There aren't 20 different versions with different pinouts. Uh-huh... Reading some more articles, I think I found the answer to my question: the SAS connector seems to be dual-sided (with conductive stripes on both sides of the plastic) while SATA ports *seem* to be the same (for backward compatibility) but only have conductive stripes on one side. Something like that :) And the SAS connector is notched so the SAS drives can't plug in to protocol-incompatible SATA-only controllers. Also some articles stated that at one time there were single-port SAS drives, so there are at least two SAS connectors after all ;) Multipathing is handled by mpxio. So I configure MPxIO, then feed the zpool create device names of multipathed aggregates, and hopefully failover should work. But can two paths work in parallel to double the bandwidth for ZFS (to single disk)? Or this depends on particular chips and drivers? Rationale: I've read a review article of an enterprise SSD which was sold back in SAS1 times (3Gbit/s) but performed at over 500Mbyte/s - and the reviewers used parallel MPIO with an LSI 9211-4i card in order to have so much actual bandwidth. Thanks, //Jim ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] JBOD recommendation for ZFS usage
On Mon, May 30, 2011 at 6:16 PM, Jim Klimov jimkli...@cos.ru wrote: Also some articles stated that at one time there were single-port SAS drives, so there are at least two SAS connectors after all ;) Nope, only one mechanical connector. A dual port cable can be used with single- or dual-ported SAS device, or with SATA drives. A single port cable can be used with a single- or dual-ported SAS device (although it will only use one port) or with a SATA drive. A SATA cable can be used with a SATA device. -B -- Brandon High : bh...@freaks.com ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] JBOD recommendation for ZFS usage
On May 30, 2011, at 6:16 PM, Jim Klimov wrote: Multipathing is handled by mpxio. So I configure MPxIO, then feed the zpool create device names of multipathed aggregates, and hopefully failover should work. Yes, it is that simple :-) But can two paths work in parallel to double the bandwidth for ZFS (to single disk)? Or this depends on particular chips and drivers? It depends more on the end-points than the stuff in the middle. Rationale: I've read a review article of an enterprise SSD which was sold back in SAS1 times (3Gbit/s) but performed at over 500Mbyte/s - and the reviewers used parallel MPIO with an LSI 9211-4i card in order to have so much actual bandwidth. The highest I've seen to a single SAS disk is 780 MB/sec sustained. Obviously, that wasn't a HDD :-) -- richard ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
