Re: [zfs-discuss] Understanding SAS/SATA Backplanes and Connectivity
Thanks Will, I thought it might be an i2c interface port to the psu, but obviously much simpler. I'll probably use a small picaxe micro, since I have a few here & have used them before. I used them to 'translate' the replacement fans clock pulse to what the monitoring circuit needed in a few V240 PSU's. Much cheaper than replacing the whole psu due to poor fan lifespan. Mark. -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Understanding SAS/SATA Backplanes and Connectivity
On Wed, Jan 6, 2010 at 20:41, Mark Bennett wrote: > Will, > > sorry for picking an old thread, That's okay---I liked this thread ;) > but you mentioned a psu monitor to supplement the CSE-PTJBOD-CB1. > I have two of these and am interested in your design. > Oddly, the LSI backplane chipset supports 2 x i2c busses that Supermicro > didn't make use of for monitoring the psu's. Okay, here goes. Basically what I found is that the 4-pin cable which comes out of the power supply gives 5 volts (at a very low amperage) when both power supplies are present, and nothing otherwise (on either the leftmost or two rightmost pins; I don't remember and it depends on which way you define as up anyways. Dig out the voltmeter and plug and unplug power supplies until you find it). So the idea is to use that tiny supply to turn on a transistor, and send high-current (well, relatively) 5v from a normal Molex connector to operate a relay. This relay is connected in series with a serial cable, and the machine on the other end runs a small daemon which writes some stuff to the serial port (in this case, a multi-port serial interface left over from our SPARC boxes) and waits to see if it gets stuff back. The source code that I use (which you should probably rewrite, now that I look at it) is attached. The whole thing is a hack, but I can say it works fine in production. I can also draw a simple schematic and recommend parts from Radio Shack if you would like, but I don't have access to that machine until next week and I don't remember exactly how it went. Ask your friendly neighborhood EE and you should be able to get there quickly. Will #include #include #include #include #include #include #include #define TheString "AReallyLongStringThatJustGoesOnAndOn" #define TheStringLength strlen(TheString) #define NAGIOS_UNKN -1 #define NAGIOS_OK0 #define NAGIOS_WARN 1 #define NAGIOS_CRIT 2 int main(int argc, char *argv[]) { if (argc < 2) { fprintf(stderr, "Usage: %s \n", argv[0]); exit(NAGIOS_UNKN); } char *ttyName = argv[1]; int fdread = open(ttyName, O_NONBLOCK); if (fdread < 0) { fprintf(stderr, "Error opening %s for read: %s\n", ttyName, strerror(errno)); exit(NAGIOS_UNKN); } int fdwrite = open(ttyName, O_WRONLY); if (fdwrite < 0) { fprintf(stderr, "Error opening %s for write: %s\n", ttyName, strerror(errno)); exit(NAGIOS_UNKN); } struct timeval otimeout, timeout; otimeout.tv_sec = 2; otimeout.tv_usec = 0; // two seconds char temp[64]; #ifdef _LOOPMODE int count = 0; #endif fd_set fdr; FD_ZERO(&fdr); FD_SET(fdread, &fdr); timeout = otimeout; int fdsChanged = select(FD_SETSIZE, &fdr, NULL, NULL, &timeout); if (fdsChanged > 0) read(fdread, temp, 63); // Read anything that might be buffered and discard it outer: #ifdef _LOOPMODE while (1) #endif { FD_SET(fdread, &fdr); // Not sure why this is needed here... int bytesw = write(fdwrite, TheString, TheStringLength); while (bytesw > 0) { timeout = otimeout; fdsChanged = select(FD_SETSIZE, &fdr, NULL, NULL, &timeout); if (fdsChanged == 0) { #ifdef _LOOPMODE fprintf(stderr, "Error reading (try %d): timeout!\n", count); count++; if (count > 5) { fprintf(stderr, "Box is down!\n"); } goto outer; #else printf("Power supply down on Supermicro chassis\n"); exit(NAGIOS_CRIT); #endif } else { #ifdef _LOOPMODE count = 0; #else printf("Supermicro PSU okay\n"); exit(NAGIOS_OK); #endif } int bytesr = read(fdread, &temp, bytesw); if (bytesr < 0) goto outer; temp[bytesr] = '\0'; bytesw -= bytesr; } } } ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Understanding SAS/SATA Backplanes and Connectivity
Will, sorry for picking an old thread, but you mentioned a psu monitor to supplement the CSE-PTJBOD-CB1. I have two of these and am interested in your design. Oddly, the LSI backplane chipset supports 2 x i2c busses that Supermicro didn't make use of for monitoring the psu's. Mark. -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Understanding SAS/SATA Backplanes and Connectivity
On Fri, 17 Jul 2009 14:16:32 -0400 Miles Nordin wrote: > > "rl" == Rob Logan writes: > > rl> Is there some magic that load balances the 4 SAS ports as this > rl> shows up as one "scsi-bus"? > > The LSI card is not SATA framework. I've the impression drive > enumeration and topology is handled by the proprietary firmware on the > card, so it's likely there isn't any explicit support for SAS > expanders inside solaris's binary mpt driver at all. There kinda is - mpt(7d) detects SAS expanders as SCSI Enclosure Services devices (which is what the spec says), and passes the enumeration off to ses(7d) or sgen(7d), depending on what you've got as a device alias for scsiclass,0d. We also (in NV since build 81, S10 Update 6) detect and correcly handle Serial Management Protocol instances, which SAS expanders hook into. The SAS HBA chip passes SMP frames to and from the expander. > If you have x86 > I think you can explore topology using the bootup Blue Screens of > Setup, but I don't have anything with SAS expander to test it. Yes, that's correct, the bluescreenofsetup allows you to do some minimal viewing of the config. > I think the SAS standard itself has a concept of ``wide ports'' like > infiniband or PCIe, so I would speculate the 4 pairs are treated as > lanes rather than ports. mpt(7d) bundles the phys and only shows one controller for internal and one controller for external connections - on a physical hba basis. cheers, James C. McPherson -- Senior Kernel Software Engineer, Solaris Sun Microsystems http://blogs.sun.com/jmcp http://www.jmcp.homeunix.com/blog Kernel Conference Australia - http://au.sun.com/sunnews/events/2009/kernel ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Understanding SAS/SATA Backplanes and Connectivity
asher...@versature.com said: > And, on that subject, is there truly a difference between Seagate's line-up > of 7200 RPM drives? They seem to now have a bunch: > . . . > Other manufacturers seem to have similar lineups. Is the difference going to > matter to me when putting a mess of them into a SAS JBOD with an expander? There are differences even within the lineup of Sun-supplied SATA drives. Some support multipathing, and some do not. Even some that are reported (in Sun docs) to support it, do not. http://opensolaris.org/jive/thread.jspa?threadID=107049&tstart=30 http://opensolaris.org/jive/thread.jspa?threadID=107057&tstart=15 Regards, Marion ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Understanding SAS/SATA Backplanes and Connectivity
On Jul 21, 2009, at 6:25 AM, F. Wessels wrote: So to wrap it up. According to Will, a supermicro chassis using a single lsi expander connected to sata disks can utilize the wide sas port between hba and the chassis. (like a J4500 Richard mentioned. How much I like these systems (thumper etc), they're way out of my budget.) Will did see more throughput than a single link could handle. But due to insufficient disks no more i/o could be generated to better demonstrate the availability off the wide sas port. I assume the J4500 is using dual expanders. Richard, can you confirm that the J4500 is using an active/active sata mux per drive to allow failover between expanders? Yes, this is seen in Figure 1-1 of the Sun Storage J4500 Array System Overview manual. Look for the Active-Active MUX (AAMUX). http://dlc.sun.com/pdf/820-3163-11/820-3163-11.pdf Aside from this can you also confirm that a wide sas link between a hba and a single expander can be fully utilized when using sata drives? To rephrase, utilization is independent of the disk type (sas or sata) It is a simple architecture. Think of expanders (and AAMUXes for this case) as switches, and it might seem more familiar. For a dual-port SAS disk, that Sun only sells in 2.5" form factor (IIRC), you don't need the AAMUX. For example, the J4400 is basically the same architecture as the J4500, except that it accepts SAS and SATA disks. Since there is a mix, it is not feasible to put the AAMUX on the system board, so they are placed in the disk carrier. If you look at the exploded part numbers, you'll see a SATA interposer card (PN 570-1181). http://sunsolve.sun.com/handbook_pub/validateUser.do?target=Systems/J4400/components -- richard All disclaimers apply. Regards Frederik -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Understanding SAS/SATA Backplanes and Connectivity
On 21-Jul-09, at 9:25 , F. Wessels wrote: So to wrap it up. According to Will, a supermicro chassis using a single lsi expander connected to sata disks can utilize the wide sas port between hba and the chassis. (like a J4500 Richard mentioned. How much I like these systems (thumper etc), they're way out of my budget.) Will did see more throughput than a single link could handle. But due to insufficient disks no more i/o could be generated to better demonstrate the availability off the wide sas port. I assume the J4500 is using dual expanders. Richard, can you confirm that the J4500 is using an active/active sata mux per drive to allow failover between expanders? Aside from this can you also confirm that a wide sas link between a hba and a single expander can be fully utilized when using sata drives? To rephrase, utilization is independent of the disk type (sas or sata) The J series JBODs aren't overly expensive, it's the darn drives for them that break the budget. And, on that subject, is there truly a difference between Seagate's line-up of 7200 RPM drives? They seem to now have a bunch: Model Bus CapacitySKU Cost (USD) 7200.12 SATA 3.0Gb/s1TB ST31000528AS$90 ES.2SATA 3.0Gb/s1TB ST31000340NS$158 ES.2SAS 3Gb/s 1TB ST31000640SS$215 Other manufacturers seem to have similar lineups. Is the difference going to matter to me when putting a mess of them into a SAS JBOD with an expander? Thanks for everyone's great feedback, this thread has been highly educating. A. -- Adam Sherman CTO, Versature Corp. Tel: +1.877.498.3772 x113 ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Understanding SAS/SATA Backplanes and Connectivity
So to wrap it up. According to Will, a supermicro chassis using a single lsi expander connected to sata disks can utilize the wide sas port between hba and the chassis. (like a J4500 Richard mentioned. How much I like these systems (thumper etc), they're way out of my budget.) Will did see more throughput than a single link could handle. But due to insufficient disks no more i/o could be generated to better demonstrate the availability off the wide sas port. I assume the J4500 is using dual expanders. Richard, can you confirm that the J4500 is using an active/active sata mux per drive to allow failover between expanders? Aside from this can you also confirm that a wide sas link between a hba and a single expander can be fully utilized when using sata drives? To rephrase, utilization is independent of the disk type (sas or sata) All disclaimers apply. Regards Frederik -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Understanding SAS/SATA Backplanes and Connectivity
F. Wessels wrote: Also in reply to the previous email by Will. Can anyone shed more light on the combination lsi sas hba , the lsisasx36 expander chip (or it's relatives) and sata disks. I'm investigating a migration from discrete channels (like in the thumper) to a multiplexed solution via a sas expander. You are describing a Sun J4500, a thumper with SAS expanders instead of a computer. http://www.sun.com/storage/disk_systems/expansion/4500/ -- richard ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Understanding SAS/SATA Backplanes and Connectivity
On Mon, Jul 20, 2009 at 05:44, F. Wessels wrote: > Also in reply to the previous email by Will. > > Can anyone shed more light on the combination lsi sas hba , the lsisasx36 > expander chip (or it's relatives) and sata disks. > I'm investigating a migration from discrete channels (like in the thumper) to > a multiplexed solution via a sas expander. > I'm aware that there's a penalty but for my workload that's acceptable. Only > when a wide sas port (4 links) is used, which should result in a theoretical > throughput of 1.2GB/s. But Will reported only 300MB/s, which corresponds to a > single link. That's not what I intended to report. I saw 400 MB/s of write I/O to my six-disk mirrored zpool, and a little over 300 MB/s read. However, the write I/O is reported as half that because it's a mirrored setup, so the userspace app writes 200 MB in a second and the disks have to write that twice in the same period. In short, I have seen more than 300 MB/s over a SAS expander. I just don't have enough non-production disks to show much more than that. Will ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Understanding SAS/SATA Backplanes and Connectivity
Also in reply to the previous email by Will. Can anyone shed more light on the combination lsi sas hba , the lsisasx36 expander chip (or it's relatives) and sata disks. I'm investigating a migration from discrete channels (like in the thumper) to a multiplexed solution via a sas expander. I'm aware that there's a penalty but for my workload that's acceptable. Only when a wide sas port (4 links) is used, which should result in a theoretical throughput of 1.2GB/s. But Will reported only 300MB/s, which corresponds to a single link. >From the limited information available regarding the lsi expanders, I found >that support for the SataTunnelProtocol is present. From the expander to the >hba only sas packets will go and therefore I see obstacle for not using a wide >port. Can this be a cable problem or a hba problem or firmware? Perhaps readers would like to share their experience with a similar setup and what setup they used. What's the throughput when only using sas disks? Regards, Frederik -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Understanding SAS/SATA Backplanes and Connectivity
> "rl" == Rob Logan writes: rl> Is there some magic that load balances the 4 SAS ports as this rl> shows up as one "scsi-bus"? The LSI card is not SATA framework. I've the impression drive enumeration and topology is handled by the proprietary firmware on the card, so it's likely there isn't any explicit support for SAS expanders inside solaris's binary mpt driver at all. If you have x86 I think you can explore topology using the bootup Blue Screens of Setup, but I don't have anything with SAS expander to test it. I think the SAS standard itself has a concept of ``wide ports'' like infiniband or PCIe, so I would speculate the 4 pairs are treated as lanes rather than ports. pgpUBrpiAUhC1.pgp Description: PGP signature ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Understanding SAS/SATA Backplanes and Connectivity
On Fri, Jul 17, 2009 at 11:43, Adam Sherman wrote: > On 17-Jul-09, at 1:45 , Will Murnane wrote: >>> >>> I'm looking at the LSI SAS3801X because it seems to be what Sun OEMs for >>> my >>> X4100s: >> >> If you're given the choice (i.e., you have the M2 revision), PCI >> Express is probably the bus to go with. It's basically the same card, >> but on a faster bus. But there's nothing wrong with the PCI-X >> version. > > I have a stack of the original X4100s. Ah, okay. PCI-X it is. >>> $280 or so, looks like. Might be overkill for me though. >> >> The 3442X-R is a little cheaper: $205 from Provantage. >> http://www.provantage.com/lsi-logic-lsi00164~7LSIG06K.htm > > > I don't get it, why is that one cheaper than: > > http://www.provantage.com/lsi-logic-lsi00124~7LSIG03W.htm When I understand LSI's pricing, I will let you know. It's got a different connector (8470 instead of 8088) on the outside, and one internal port. On the other hand, that card only gets you one external port. If you plan to have more than one JBOD per host, the 3801 makes more sense. Will ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Understanding SAS/SATA Backplanes and Connectivity
On 17-Jul-09, at 1:45 , Will Murnane wrote: I'm looking at the LSI SAS3801X because it seems to be what Sun OEMs for my X4100s: If you're given the choice (i.e., you have the M2 revision), PCI Express is probably the bus to go with. It's basically the same card, but on a faster bus. But there's nothing wrong with the PCI-X version. I have a stack of the original X4100s. $280 or so, looks like. Might be overkill for me though. The 3442X-R is a little cheaper: $205 from Provantage. http://www.provantage.com/lsi-logic-lsi00164~7LSIG06K.htm I don't get it, why is that one cheaper than: http://www.provantage.com/lsi-logic-lsi00124~7LSIG03W.htm Just newer? A. -- Adam Sherman CTO, Versature Corp. Tel: +1.877.498.3772 x113 ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Understanding SAS/SATA Backplanes and Connectivity
On Thu, Jul 16, 2009 at 21:30, Rob Logan wrote: >> c4 scsi-bus connected configured >> unknown >> c4::dsk/c4t15d0 disk connected configured >> unknown > : >> c4::dsk/c4t33d0 disk connected configured >> unknown >> c4::es/ses0 ESI connected configured >> unknown > > thanks! so SATA disks show up JBOD in IT mode.. Is there some magic that > load balances the 4 SAS ports as this shows up as one "scsi-bus"? Hypothetically, yes. In practical terms, though, I've seen more than 300 MB/s of I/O over it: capacity operationsbandwidth pool used avail read write read write --- - - - - - - data11.06T 1.21T 1 1.61K 2.49K 200M mirror 460G 236G 0522 1.15K 63.8M c4t18d0 - - 0518 6.38K 63.6M c4t21d0 - - 0518 12.8K 63.8M mirror 467G 229G 0533306 64.8M c4t23d0 - - 0523 6.38K 64.3M c4t25d0 - - 0529 0 65.0M mirror 153G 775G 0597 1.05K 71.8M c4t20d0 - - 0589 12.8K 72.5M c4t22d0 - - 0584 0 71.8M --- - - - - - - Note that the pool is only doing 200 MB/s, but the individual devices are doing a total of 400 MB/s. It's not possible to put more than 300 MB/s into or out of a single device, so there's no "link aggregation" to worry about. Will ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Understanding SAS/SATA Backplanes and Connectivity
On Thu, Jul 16, 2009 at 21:35, Adam Sherman wrote: > I'm looking at the LSI SAS3801X because it seems to be what Sun OEMs for my > X4100s: If you're given the choice (i.e., you have the M2 revision), PCI Express is probably the bus to go with. It's basically the same card, but on a faster bus. But there's nothing wrong with the PCI-X version. http://www.lsi.com/storage_home/products_home/host_bus_adapters/sas_hbas/lsisas3801e/index.html > $280 or so, looks like. Might be overkill for me though. The 3442X-R is a little cheaper: $205 from Provantage. http://www.provantage.com/lsi-logic-lsi00164~7LSIG06K.htm Will ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Understanding SAS/SATA Backplanes and Connectivity
On 16-Jul-09, at 21:17 , Will Murnane wrote: Good to hear. What HBA(s) are you using against it? LSI 3442E-R. It's connected through a Supermicro cable, CBL-0168L, so it can be attached via an external cable. I'm looking at the LSI SAS3801X because it seems to be what Sun OEMs for my X4100s: http://sunsolve.sun.com/handbook_private/validateUser.do?target=Devices/SCSI/SCSI_PCIX_SAS_SATA_HBA $280 or so, looks like. Might be overkill for me though. A. -- Adam Sherman CTO, Versature Corp. Tel: +1.877.498.3772 x113 ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Understanding SAS/SATA Backplanes and Connectivity
> c4 scsi-bus connectedconfigured unknown > c4::dsk/c4t15d0disk connectedconfigured unknown : > c4::dsk/c4t33d0disk connectedconfigured unknown > c4::es/ses0ESI connectedconfigured unknown thanks! so SATA disks show up JBOD in IT mode.. Is there some magic that load balances the 4 SAS ports as this shows up as one "scsi-bus"? ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Understanding SAS/SATA Backplanes and Connectivity
On Thu, Jul 16, 2009 at 21:16, Rob Logan wrote: > I'm confused, I though expanders only worked with SAS disk, and SATA disks > took an entire SAS port. could someone post an output showing more than 4 > SATA > drives across one InfiniBand cable (4 SAS ports) > > 2 % cfgadm | grep sata > sata1/0::dsk/c9t0d0 cd/dvd connected configured ok > sata1/1::dsk/c9t1d0 disk connected configured ok > sata1/2::dsk/c9t2d0 disk connected configured ok > sata1/3 sata-port empty unconfigured ok > sata1/4::dsk/c9t4d0 disk connected configured ok > sata1/5 sata-port empty unconfigured ok > sata2/0::dsk/c7t0d0 disk connected configured ok > sata2/1::dsk/c7t1d0 disk connected configured ok > sata2/2::dsk/c7t2d0 disk connected configured ok > sata2/3 sata-port empty unconfigured ok > sata2/4::dsk/c7t4d0 disk connected configured ok > sata2/5::dsk/c7t5d0 disk connected configured ok > sata2/6 sata-port empty unconfigured ok > sata2/7 sata-port empty unconfigured ok > sata3/0::dsk/c8t0d0 disk connected configured ok > sata3/1::dsk/c8t1d0 disk connected configured ok > sata3/2::dsk/c8t2d0 disk connected configured ok > sata3/3 sata-port empty unconfigured ok > sata3/4::dsk/c8t4d0 disk connected configured ok > sata3/5::dsk/c8t5d0 disk connected configured ok > sata3/6 sata-port empty unconfigured ok > sata3/7 sata-port empty unconfigured ok Here's the relevant part of cfgadm -al on our machine. The disks are all sata. c4 scsi-bus connectedconfigured unknown c4::dsk/c4t15d0disk connectedconfigured unknown c4::dsk/c4t17d0disk connectedconfigured unknown c4::dsk/c4t18d0disk connectedconfigured unknown c4::dsk/c4t19d0disk connectedconfigured unknown c4::dsk/c4t20d0disk connectedconfigured unknown c4::dsk/c4t21d0disk connectedconfigured unknown c4::dsk/c4t22d0disk connectedconfigured unknown c4::dsk/c4t23d0disk connectedconfigured unknown c4::dsk/c4t24d0disk connectedconfigured unknown c4::dsk/c4t25d0disk connectedconfigured unknown c4::dsk/c4t26d0disk connectedconfigured unknown c4::dsk/c4t27d0disk connectedconfigured unknown c4::dsk/c4t28d0disk connectedconfigured unknown c4::dsk/c4t29d0disk connectedconfigured unknown c4::dsk/c4t30d0disk connectedconfigured unknown c4::dsk/c4t31d0disk connectedconfigured unknown c4::dsk/c4t32d0disk connectedconfigured unknown c4::dsk/c4t33d0disk connectedconfigured unknown c4::es/ses0ESI connectedconfigured unknown Will ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Understanding SAS/SATA Backplanes and Connectivity
On Thu, Jul 16, 2009 at 20:20, Adam Sherman wrote: > Ever seen/read about anyone use this kind of setup for HA clustering? I'm > getting ideas about Open HA/Solaris Cluster on top of this setup with two > systems connecting, that would rock! It's possible that this would work with homogeneous hardware, but I tried with another LSI-based expander and SATA disks, and had no luck. Perhaps SAS is necessary? >> We have a SC846E1 at work; it's the 24-disk, 4u version of the 826e1. >> It's working quite nicely as a SATA JBOD enclosure. We'll probably be >> buying another in the coming year to have more capacity. > > Good to hear. What HBA(s) are you using against it? LSI 3442E-R. It's connected through a Supermicro cable, CBL-0168L, so it can be attached via an external cable. There's a card needed, CSE-PTJBOD-CB1, that allows the case to run without a motherboard in it. There's no monitoring for the power supplies, but I built one for it; I can provide schematics and suggested part numbers if you're interested. >> We have a SC846E1 at work; it's the 24-disk, 4u version of the 826e1. >> It's working quite nicely as a SATA JBOD enclosure. We'll probably be >> buying another in the coming year to have more capacity. > > > I should also ask: any other solutions I should have a look at to get >=12 > SATA disks externally attached to my systems? This was the best solution we found for the money. The 826 is about $750, while the 846 is $1100 shipped (wiredzone.com). Per disk, the 846 is almost $20 cheaper. If you only care for 12 disks, then one might as well not spend the extra money, but if there's potential for expansion it's a wise investment. Will ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Understanding SAS/SATA Backplanes and Connectivity
>> We have a SC846E1 at work; it's the 24-disk, 4u version of the 826e1. >> It's working quite nicely as a SATA JBOD enclosure. > use the LSI SAS 3442e which also gives you an external SAS port. I'm confused, I though expanders only worked with SAS disk, and SATA disks took an entire SAS port. could someone post an output showing more than 4 SATA drives across one InfiniBand cable (4 SAS ports) 2 % cfgadm | grep sata sata1/0::dsk/c9t0d0cd/dvd connectedconfigured ok sata1/1::dsk/c9t1d0disk connectedconfigured ok sata1/2::dsk/c9t2d0disk connectedconfigured ok sata1/3sata-portemptyunconfigured ok sata1/4::dsk/c9t4d0disk connectedconfigured ok sata1/5sata-portemptyunconfigured ok sata2/0::dsk/c7t0d0disk connectedconfigured ok sata2/1::dsk/c7t1d0disk connectedconfigured ok sata2/2::dsk/c7t2d0disk connectedconfigured ok sata2/3sata-portemptyunconfigured ok sata2/4::dsk/c7t4d0disk connectedconfigured ok sata2/5::dsk/c7t5d0disk connectedconfigured ok sata2/6sata-portemptyunconfigured ok sata2/7sata-portemptyunconfigured ok sata3/0::dsk/c8t0d0disk connectedconfigured ok sata3/1::dsk/c8t1d0disk connectedconfigured ok sata3/2::dsk/c8t2d0disk connectedconfigured ok sata3/3sata-portemptyunconfigured ok sata3/4::dsk/c8t4d0disk connectedconfigured ok sata3/5::dsk/c8t5d0disk connectedconfigured ok sata3/6sata-portemptyunconfigured ok sata3/7sata-portemptyunconfigured ok ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Understanding SAS/SATA Backplanes and Connectivity
On Thu, 16 Jul 2009, Adam Sherman wrote: I should also ask: any other solutions I should have a look at to get >=12 SATA disks externally attached to my systems? Depending on how much failure resiliancy you want and how you plan to configure your pool, you may be better off with two independent disk trays with 12 disks each. For example, if you were to use mirrors, you could split the mirrors across the disk trays. If one tray fails, then your system still works. If you are planning to use raidz1 or raidz2 then there is likely no benefit to going with two smaller trays. Bob -- Bob Friesenhahn bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/ ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Understanding SAS/SATA Backplanes and Connectivity
On 16-Jul-09, at 18:01 , Will Murnane wrote: We have a SC846E1 at work; it's the 24-disk, 4u version of the 826e1. It's working quite nicely as a SATA JBOD enclosure. We'll probably be buying another in the coming year to have more capacity. I should also ask: any other solutions I should have a look at to get >=12 SATA disks externally attached to my systems? Thanks! A. -- Adam Sherman CTO, Versature Corp. Tel: +1.877.498.3772 x113 ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Understanding SAS/SATA Backplanes and Connectivity
On 16-Jul-09, at 20:52 , James C. McPherson wrote: Another thought in the same vein, I notice many of these systems support "SES-2" for management. Does this do anything useful under Solaris? We've got some integration between FMA and SES devices which allows us to to some management tasks. So that would allow FMA to detect SATA disk failures then? libtopo, libscsi and libses are the main methods of getting that information out. For an example outside FMA, you could have a look into the ses/sgen plugin from pluggable fwflash. Is there anything you're specifically interested in wrt management uses of SES? I'm really just exploring. Where can I read about how FMA is going to help with failures in my setup? Thanks, A. -- Adam Sherman CTO, Versature Corp. Tel: +1.877.498.3772 x113 ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Understanding SAS/SATA Backplanes and Connectivity
On Thu, 16 Jul 2009 20:26:17 -0400 Adam Sherman wrote: > Another thought in the same vein, I notice many of these systems > support "SES-2" for management. Does this do anything useful under > Solaris? We've got some integration between FMA and SES devices which allows us to to some management tasks. libtopo, libscsi and libses are the main methods of getting that information out. For an example outside FMA, you could have a look into the ses/sgen plugin from pluggable fwflash. Is there anything you're specifically interested in wrt management uses of SES? thanks, James C. McPherson -- Senior Kernel Software Engineer, Solaris Sun Microsystems http://blogs.sun.com/jmcp http://www.jmcp.homeunix.com/blog Kernel Conference Australia - http://au.sun.com/sunnews/events/2009/kernel ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Understanding SAS/SATA Backplanes and Connectivity
> > > We have a SC846E1 at work; it's the 24-disk, 4u > version of the 826e1. > > It's working quite nicely as a SATA JBOD enclosure. > We'll probably be > buying another in the coming year to have more > capacity. > Good to hear. What HBA(s) are you using against it? > I've got one too and it works great. I use the LSI SAS 3442e which also gives you an external SAS port. You don't need a fancy HBA with onboard RAID. Configure to IT mode. -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Understanding SAS/SATA Backplanes and Connectivity
Another thought in the same vein, I notice many of these systems support "SES-2" for management. Does this do anything useful under Solaris? Sorry for these questions, I seem to be having a tough time locating relevant information on the web. Thanks, A. -- Adam Sherman CTO, Versature Corp. Tel: +1.877.498.3772 x113 ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Understanding SAS/SATA Backplanes and Connectivity
On 16-Jul-09, at 18:01 , Will Murnane wrote: The "direct attached" backplane is right out. This means that each drive has its own individual sata port, meaning you'd need three SAS wide ports just to connect the drives. The single-expander version has one LSI SAS expander, which connects to all the drives and has two "upstream" ports. This means you plug in one or two servers directly, and they can both see all the disks. I've only tested this with one-server configurations. It also has one "downstream" port which you could use to daisy-chain more expanders (i.e., more 826/846 cases) onto the same server. That makes things a heck of a lot clearer, thank you very much for taking the time to explain! Ever seen/read about anyone use this kind of setup for HA clustering? I'm getting ideas about Open HA/Solaris Cluster on top of this setup with two systems connecting, that would rock! We have a SC846E1 at work; it's the 24-disk, 4u version of the 826e1. It's working quite nicely as a SATA JBOD enclosure. We'll probably be buying another in the coming year to have more capacity. Good to hear. What HBA(s) are you using against it? Thanks for pointing to relevant documentation. The manual for the Supermicro cases [1, 2] does a pretty good job IMO explaining the different options. See page D-14 and on in the 826 manual, or page D-11 and on in the 846 manual. I'll read though that, thanks for the detailed pointers. A. -- Adam Sherman CTO, Versature Corp. Tel: +1.877.498.3772 x113 ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Understanding SAS/SATA Backplanes and Connectivity
On Thu, Jul 16, 2009 at 17:02, Adam Sherman wrote: > Hello All, > > I'm just starting to think about building some mass-storage arrays and am > looking to better understand some of the components involved. > > For example, the Supermicro SC826 series of systems is available with three > backplanes: > > 1. SAS / SATA Expander Backplane with single LSI SASX28 Expander Chip > 2. SAS / SATA Expander Backplane with dual LSI SASX28 Expander Chips > 3. SAS / SATA Direct Attached Backplane > > Assuming I am using this an external array, connected to a server via SAS, > how do these fit into my topology? Expander, dual-expanders and no expander? > Huh? The "direct attached" backplane is right out. This means that each drive has its own individual sata port, meaning you'd need three SAS wide ports just to connect the drives. The single-expander version has one LSI SAS expander, which connects to all the drives and has two "upstream" ports. This means you plug in one or two servers directly, and they can both see all the disks. I've only tested this with one-server configurations. It also has one "downstream" port which you could use to daisy-chain more expanders (i.e., more 826/846 cases) onto the same server. We have a SC846E1 at work; it's the 24-disk, 4u version of the 826e1. It's working quite nicely as a SATA JBOD enclosure. We'll probably be buying another in the coming year to have more capacity. The dual-expander version has two LSI SAS expanders. You need dual-port SAS drives (not SATA). This lets you have two paths all the way to each drive; even if one expander fails (this seems pretty unlikely to me, but if you're shooting for many nines it's worth considering) you still have access to the disks. > Thanks for pointing to relevant documentation. The manual for the Supermicro cases [1, 2] does a pretty good job IMO explaining the different options. See page D-14 and on in the 826 manual, or page D-11 and on in the 846 manual. Will [1]: http://supermicro.com/manuals/chassis/2U/SC826.pdf [2]: http://supermicro.com/manuals/chassis/tower/SC846.pdf ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss