Re: [zfs-discuss] Understanding SAS/SATA Backplanes and Connectivity
Thanks Will, I thought it might be an i2c interface port to the psu, but obviously much simpler. I'll probably use a small picaxe micro, since I have a few here have used them before. I used them to 'translate' the replacement fans clock pulse to what the monitoring circuit needed in a few V240 PSU's. Much cheaper than replacing the whole psu due to poor fan lifespan. Mark. -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Understanding SAS/SATA Backplanes and Connectivity
Will, sorry for picking an old thread, but you mentioned a psu monitor to supplement the CSE-PTJBOD-CB1. I have two of these and am interested in your design. Oddly, the LSI backplane chipset supports 2 x i2c busses that Supermicro didn't make use of for monitoring the psu's. Mark. -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Understanding SAS/SATA Backplanes and Connectivity
On Wed, Jan 6, 2010 at 20:41, Mark Bennett mark.benn...@public.co.nz wrote:
Will,
sorry for picking an old thread,
That's okay---I liked this thread ;)
but you mentioned a psu monitor to supplement the CSE-PTJBOD-CB1.
I have two of these and am interested in your design.
Oddly, the LSI backplane chipset supports 2 x i2c busses that Supermicro
didn't make use of for monitoring the psu's.
Okay, here goes. Basically what I found is that the 4-pin cable which
comes out of the power supply gives 5 volts (at a very low amperage)
when both power supplies are present, and nothing otherwise (on either
the leftmost or two rightmost pins; I don't remember and it depends on
which way you define as up anyways. Dig out the voltmeter and plug
and unplug power supplies until you find it). So the idea is to use
that tiny supply to turn on a transistor, and send high-current (well,
relatively) 5v from a normal Molex connector to operate a relay. This
relay is connected in series with a serial cable, and the machine on
the other end runs a small daemon which writes some stuff to the
serial port (in this case, a multi-port serial interface left over
from our SPARC boxes) and waits to see if it gets stuff back. The
source code that I use (which you should probably rewrite, now that I
look at it) is attached. The whole thing is a hack, but I can say it
works fine in production.
I can also draw a simple schematic and recommend parts from Radio
Shack if you would like, but I don't have access to that machine until
next week and I don't remember exactly how it went. Ask your friendly
neighborhood EE and you should be able to get there quickly.
Will
#include sys/select.h
#include fcntl.h
#include stdio.h
#include stdlib.h
#include unistd.h
#include string.h
#include errno.h
#define TheString AReallyLongStringThatJustGoesOnAndOn
#define TheStringLength strlen(TheString)
#define NAGIOS_UNKN -1
#define NAGIOS_OK0
#define NAGIOS_WARN 1
#define NAGIOS_CRIT 2
int main(int argc, char *argv[])
{
if (argc 2)
{
fprintf(stderr, Usage: %s tty\n, argv[0]);
exit(NAGIOS_UNKN);
}
char *ttyName = argv[1];
int fdread = open(ttyName, O_NONBLOCK);
if (fdread 0)
{
fprintf(stderr, Error opening %s for read: %s\n, ttyName, strerror(errno));
exit(NAGIOS_UNKN);
}
int fdwrite = open(ttyName, O_WRONLY);
if (fdwrite 0)
{
fprintf(stderr, Error opening %s for write: %s\n, ttyName, strerror(errno));
exit(NAGIOS_UNKN);
}
struct timeval otimeout, timeout;
otimeout.tv_sec = 2;
otimeout.tv_usec = 0; // two seconds
char temp[64];
#ifdef _LOOPMODE
int count = 0;
#endif
fd_set fdr;
FD_ZERO(fdr);
FD_SET(fdread, fdr);
timeout = otimeout;
int fdsChanged = select(FD_SETSIZE, fdr, NULL, NULL, timeout);
if (fdsChanged 0)
read(fdread, temp, 63); // Read anything that might be buffered and discard it
outer:
#ifdef _LOOPMODE
while (1)
#endif
{
FD_SET(fdread, fdr); // Not sure why this is needed here...
int bytesw = write(fdwrite, TheString, TheStringLength);
while (bytesw 0)
{
timeout = otimeout;
fdsChanged = select(FD_SETSIZE, fdr, NULL, NULL, timeout);
if (fdsChanged == 0)
{
#ifdef _LOOPMODE
fprintf(stderr, Error reading (try %d): timeout!\n, count);
count++;
if (count 5)
{
fprintf(stderr, Box is down!\n);
}
goto outer;
#else
printf(Power supply down on Supermicro chassis\n);
exit(NAGIOS_CRIT);
#endif
} else {
#ifdef _LOOPMODE
count = 0;
#else
printf(Supermicro PSU okay\n);
exit(NAGIOS_OK);
#endif
}
int bytesr = read(fdread, temp, bytesw);
if (bytesr 0)
goto outer;
temp[bytesr] = '\0';
bytesw -= bytesr;
}
}
}
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Understanding SAS/SATA Backplanes and Connectivity
On Fri, 17 Jul 2009 14:16:32 -0400 Miles Nordin car...@ivy.net wrote: rl == Rob Logan r...@logan.com writes: rl Is there some magic that load balances the 4 SAS ports as this rl shows up as one scsi-bus? The LSI card is not SATA framework. I've the impression drive enumeration and topology is handled by the proprietary firmware on the card, so it's likely there isn't any explicit support for SAS expanders inside solaris's binary mpt driver at all. There kinda is - mpt(7d) detects SAS expanders as SCSI Enclosure Services devices (which is what the spec says), and passes the enumeration off to ses(7d) or sgen(7d), depending on what you've got as a device alias for scsiclass,0d. We also (in NV since build 81, S10 Update 6) detect and correcly handle Serial Management Protocol instances, which SAS expanders hook into. The SAS HBA chip passes SMP frames to and from the expander. If you have x86 I think you can explore topology using the bootup Blue Screens of Setup, but I don't have anything with SAS expander to test it. Yes, that's correct, the bluescreenofsetup allows you to do some minimal viewing of the config. I think the SAS standard itself has a concept of ``wide ports'' like infiniband or PCIe, so I would speculate the 4 pairs are treated as lanes rather than ports. mpt(7d) bundles the phys and only shows one controller for internal and one controller for external connections - on a physical hba basis. cheers, James C. McPherson -- Senior Kernel Software Engineer, Solaris Sun Microsystems http://blogs.sun.com/jmcp http://www.jmcp.homeunix.com/blog Kernel Conference Australia - http://au.sun.com/sunnews/events/2009/kernel ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Understanding SAS/SATA Backplanes and Connectivity
So to wrap it up. According to Will, a supermicro chassis using a single lsi expander connected to sata disks can utilize the wide sas port between hba and the chassis. (like a J4500 Richard mentioned. How much I like these systems (thumper etc), they're way out of my budget.) Will did see more throughput than a single link could handle. But due to insufficient disks no more i/o could be generated to better demonstrate the availability off the wide sas port. I assume the J4500 is using dual expanders. Richard, can you confirm that the J4500 is using an active/active sata mux per drive to allow failover between expanders? Aside from this can you also confirm that a wide sas link between a hba and a single expander can be fully utilized when using sata drives? To rephrase, utilization is independent of the disk type (sas or sata) All disclaimers apply. Regards Frederik -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Understanding SAS/SATA Backplanes and Connectivity
On 21-Jul-09, at 9:25 , F. Wessels wrote: So to wrap it up. According to Will, a supermicro chassis using a single lsi expander connected to sata disks can utilize the wide sas port between hba and the chassis. (like a J4500 Richard mentioned. How much I like these systems (thumper etc), they're way out of my budget.) Will did see more throughput than a single link could handle. But due to insufficient disks no more i/o could be generated to better demonstrate the availability off the wide sas port. I assume the J4500 is using dual expanders. Richard, can you confirm that the J4500 is using an active/active sata mux per drive to allow failover between expanders? Aside from this can you also confirm that a wide sas link between a hba and a single expander can be fully utilized when using sata drives? To rephrase, utilization is independent of the disk type (sas or sata) The J series JBODs aren't overly expensive, it's the darn drives for them that break the budget. And, on that subject, is there truly a difference between Seagate's line-up of 7200 RPM drives? They seem to now have a bunch: Model Bus CapacitySKU Cost (USD) 7200.12 SATA 3.0Gb/s1TB ST31000528AS$90 ES.2SATA 3.0Gb/s1TB ST31000340NS$158 ES.2SAS 3Gb/s 1TB ST31000640SS$215 Other manufacturers seem to have similar lineups. Is the difference going to matter to me when putting a mess of them into a SAS JBOD with an expander? Thanks for everyone's great feedback, this thread has been highly educating. A. -- Adam Sherman CTO, Versature Corp. Tel: +1.877.498.3772 x113 ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Understanding SAS/SATA Backplanes and Connectivity
On Jul 21, 2009, at 6:25 AM, F. Wessels wrote: So to wrap it up. According to Will, a supermicro chassis using a single lsi expander connected to sata disks can utilize the wide sas port between hba and the chassis. (like a J4500 Richard mentioned. How much I like these systems (thumper etc), they're way out of my budget.) Will did see more throughput than a single link could handle. But due to insufficient disks no more i/o could be generated to better demonstrate the availability off the wide sas port. I assume the J4500 is using dual expanders. Richard, can you confirm that the J4500 is using an active/active sata mux per drive to allow failover between expanders? Yes, this is seen in Figure 1-1 of the Sun Storage J4500 Array System Overview manual. Look for the Active-Active MUX (AAMUX). http://dlc.sun.com/pdf/820-3163-11/820-3163-11.pdf Aside from this can you also confirm that a wide sas link between a hba and a single expander can be fully utilized when using sata drives? To rephrase, utilization is independent of the disk type (sas or sata) It is a simple architecture. Think of expanders (and AAMUXes for this case) as switches, and it might seem more familiar. For a dual-port SAS disk, that Sun only sells in 2.5 form factor (IIRC), you don't need the AAMUX. For example, the J4400 is basically the same architecture as the J4500, except that it accepts SAS and SATA disks. Since there is a mix, it is not feasible to put the AAMUX on the system board, so they are placed in the disk carrier. If you look at the exploded part numbers, you'll see a SATA interposer card (PN 570-1181). http://sunsolve.sun.com/handbook_pub/validateUser.do?target=Systems/J4400/components -- richard All disclaimers apply. Regards Frederik -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Understanding SAS/SATA Backplanes and Connectivity
asher...@versature.com said: And, on that subject, is there truly a difference between Seagate's line-up of 7200 RPM drives? They seem to now have a bunch: . . . Other manufacturers seem to have similar lineups. Is the difference going to matter to me when putting a mess of them into a SAS JBOD with an expander? There are differences even within the lineup of Sun-supplied SATA drives. Some support multipathing, and some do not. Even some that are reported (in Sun docs) to support it, do not. http://opensolaris.org/jive/thread.jspa?threadID=107049tstart=30 http://opensolaris.org/jive/thread.jspa?threadID=107057tstart=15 Regards, Marion ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Understanding SAS/SATA Backplanes and Connectivity
Also in reply to the previous email by Will. Can anyone shed more light on the combination lsi sas hba , the lsisasx36 expander chip (or it's relatives) and sata disks. I'm investigating a migration from discrete channels (like in the thumper) to a multiplexed solution via a sas expander. I'm aware that there's a penalty but for my workload that's acceptable. Only when a wide sas port (4 links) is used, which should result in a theoretical throughput of 1.2GB/s. But Will reported only 300MB/s, which corresponds to a single link. From the limited information available regarding the lsi expanders, I found that support for the SataTunnelProtocol is present. From the expander to the hba only sas packets will go and therefore I see obstacle for not using a wide port. Can this be a cable problem or a hba problem or firmware? Perhaps readers would like to share their experience with a similar setup and what setup they used. What's the throughput when only using sas disks? Regards, Frederik -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Understanding SAS/SATA Backplanes and Connectivity
On Mon, Jul 20, 2009 at 05:44, F. Wesselsno-re...@opensolaris.org wrote: Also in reply to the previous email by Will. Can anyone shed more light on the combination lsi sas hba , the lsisasx36 expander chip (or it's relatives) and sata disks. I'm investigating a migration from discrete channels (like in the thumper) to a multiplexed solution via a sas expander. I'm aware that there's a penalty but for my workload that's acceptable. Only when a wide sas port (4 links) is used, which should result in a theoretical throughput of 1.2GB/s. But Will reported only 300MB/s, which corresponds to a single link. That's not what I intended to report. I saw 400 MB/s of write I/O to my six-disk mirrored zpool, and a little over 300 MB/s read. However, the write I/O is reported as half that because it's a mirrored setup, so the userspace app writes 200 MB in a second and the disks have to write that twice in the same period. In short, I have seen more than 300 MB/s over a SAS expander. I just don't have enough non-production disks to show much more than that. Will ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Understanding SAS/SATA Backplanes and Connectivity
F. Wessels wrote: Also in reply to the previous email by Will. Can anyone shed more light on the combination lsi sas hba , the lsisasx36 expander chip (or it's relatives) and sata disks. I'm investigating a migration from discrete channels (like in the thumper) to a multiplexed solution via a sas expander. You are describing a Sun J4500, a thumper with SAS expanders instead of a computer. http://www.sun.com/storage/disk_systems/expansion/4500/ -- richard ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Understanding SAS/SATA Backplanes and Connectivity
On Thu, Jul 16, 2009 at 21:30, Rob Loganr...@logan.com wrote: c4 scsi-bus connected configured unknown c4::dsk/c4t15d0 disk connected configured unknown : c4::dsk/c4t33d0 disk connected configured unknown c4::es/ses0 ESI connected configured unknown thanks! so SATA disks show up JBOD in IT mode.. Is there some magic that load balances the 4 SAS ports as this shows up as one scsi-bus? Hypothetically, yes. In practical terms, though, I've seen more than 300 MB/s of I/O over it: capacity operationsbandwidth pool used avail read write read write --- - - - - - - data11.06T 1.21T 1 1.61K 2.49K 200M mirror 460G 236G 0522 1.15K 63.8M c4t18d0 - - 0518 6.38K 63.6M c4t21d0 - - 0518 12.8K 63.8M mirror 467G 229G 0533306 64.8M c4t23d0 - - 0523 6.38K 64.3M c4t25d0 - - 0529 0 65.0M mirror 153G 775G 0597 1.05K 71.8M c4t20d0 - - 0589 12.8K 72.5M c4t22d0 - - 0584 0 71.8M --- - - - - - - Note that the pool is only doing 200 MB/s, but the individual devices are doing a total of 400 MB/s. It's not possible to put more than 300 MB/s into or out of a single device, so there's no link aggregation to worry about. Will ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Understanding SAS/SATA Backplanes and Connectivity
On 17-Jul-09, at 1:45 , Will Murnane wrote: I'm looking at the LSI SAS3801X because it seems to be what Sun OEMs for my X4100s: If you're given the choice (i.e., you have the M2 revision), PCI Express is probably the bus to go with. It's basically the same card, but on a faster bus. But there's nothing wrong with the PCI-X version. I have a stack of the original X4100s. $280 or so, looks like. Might be overkill for me though. The 3442X-R is a little cheaper: $205 from Provantage. http://www.provantage.com/lsi-logic-lsi00164~7LSIG06K.htm I don't get it, why is that one cheaper than: http://www.provantage.com/lsi-logic-lsi00124~7LSIG03W.htm Just newer? A. -- Adam Sherman CTO, Versature Corp. Tel: +1.877.498.3772 x113 ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Understanding SAS/SATA Backplanes and Connectivity
On Fri, Jul 17, 2009 at 11:43, Adam Shermanasher...@versature.com wrote: On 17-Jul-09, at 1:45 , Will Murnane wrote: I'm looking at the LSI SAS3801X because it seems to be what Sun OEMs for my X4100s: If you're given the choice (i.e., you have the M2 revision), PCI Express is probably the bus to go with. It's basically the same card, but on a faster bus. But there's nothing wrong with the PCI-X version. I have a stack of the original X4100s. Ah, okay. PCI-X it is. $280 or so, looks like. Might be overkill for me though. The 3442X-R is a little cheaper: $205 from Provantage. http://www.provantage.com/lsi-logic-lsi00164~7LSIG06K.htm I don't get it, why is that one cheaper than: http://www.provantage.com/lsi-logic-lsi00124~7LSIG03W.htm When I understand LSI's pricing, I will let you know. It's got a different connector (8470 instead of 8088) on the outside, and one internal port. On the other hand, that card only gets you one external port. If you plan to have more than one JBOD per host, the 3801 makes more sense. Will ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Understanding SAS/SATA Backplanes and Connectivity
rl == Rob Logan r...@logan.com writes: rl Is there some magic that load balances the 4 SAS ports as this rl shows up as one scsi-bus? The LSI card is not SATA framework. I've the impression drive enumeration and topology is handled by the proprietary firmware on the card, so it's likely there isn't any explicit support for SAS expanders inside solaris's binary mpt driver at all. If you have x86 I think you can explore topology using the bootup Blue Screens of Setup, but I don't have anything with SAS expander to test it. I think the SAS standard itself has a concept of ``wide ports'' like infiniband or PCIe, so I would speculate the 4 pairs are treated as lanes rather than ports. pgpUBrpiAUhC1.pgp Description: PGP signature ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] Understanding SAS/SATA Backplanes and Connectivity
Hello All, I'm just starting to think about building some mass-storage arrays and am looking to better understand some of the components involved. For example, the Supermicro SC826 series of systems is available with three backplanes: 1. SAS / SATA Expander Backplane with single LSI SASX28 Expander Chip 2. SAS / SATA Expander Backplane with dual LSI SASX28 Expander Chips 3. SAS / SATA Direct Attached Backplane Assuming I am using this an external array, connected to a server via SAS, how do these fit into my topology? Expander, dual-expanders and no expander? Huh? Thanks for pointing to relevant documentation. A. -- Adam Sherman CTO, Versature Corp. Tel: +1.877.498.3772 x113 ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Understanding SAS/SATA Backplanes and Connectivity
On Thu, Jul 16, 2009 at 17:02, Adam Shermanasher...@versature.com wrote: Hello All, I'm just starting to think about building some mass-storage arrays and am looking to better understand some of the components involved. For example, the Supermicro SC826 series of systems is available with three backplanes: 1. SAS / SATA Expander Backplane with single LSI SASX28 Expander Chip 2. SAS / SATA Expander Backplane with dual LSI SASX28 Expander Chips 3. SAS / SATA Direct Attached Backplane Assuming I am using this an external array, connected to a server via SAS, how do these fit into my topology? Expander, dual-expanders and no expander? Huh? The direct attached backplane is right out. This means that each drive has its own individual sata port, meaning you'd need three SAS wide ports just to connect the drives. The single-expander version has one LSI SAS expander, which connects to all the drives and has two upstream ports. This means you plug in one or two servers directly, and they can both see all the disks. I've only tested this with one-server configurations. It also has one downstream port which you could use to daisy-chain more expanders (i.e., more 826/846 cases) onto the same server. We have a SC846E1 at work; it's the 24-disk, 4u version of the 826e1. It's working quite nicely as a SATA JBOD enclosure. We'll probably be buying another in the coming year to have more capacity. The dual-expander version has two LSI SAS expanders. You need dual-port SAS drives (not SATA). This lets you have two paths all the way to each drive; even if one expander fails (this seems pretty unlikely to me, but if you're shooting for many nines it's worth considering) you still have access to the disks. Thanks for pointing to relevant documentation. The manual for the Supermicro cases [1, 2] does a pretty good job IMO explaining the different options. See page D-14 and on in the 826 manual, or page D-11 and on in the 846 manual. Will [1]: http://supermicro.com/manuals/chassis/2U/SC826.pdf [2]: http://supermicro.com/manuals/chassis/tower/SC846.pdf ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Understanding SAS/SATA Backplanes and Connectivity
On 16-Jul-09, at 18:01 , Will Murnane wrote: The direct attached backplane is right out. This means that each drive has its own individual sata port, meaning you'd need three SAS wide ports just to connect the drives. The single-expander version has one LSI SAS expander, which connects to all the drives and has two upstream ports. This means you plug in one or two servers directly, and they can both see all the disks. I've only tested this with one-server configurations. It also has one downstream port which you could use to daisy-chain more expanders (i.e., more 826/846 cases) onto the same server. That makes things a heck of a lot clearer, thank you very much for taking the time to explain! Ever seen/read about anyone use this kind of setup for HA clustering? I'm getting ideas about Open HA/Solaris Cluster on top of this setup with two systems connecting, that would rock! We have a SC846E1 at work; it's the 24-disk, 4u version of the 826e1. It's working quite nicely as a SATA JBOD enclosure. We'll probably be buying another in the coming year to have more capacity. Good to hear. What HBA(s) are you using against it? Thanks for pointing to relevant documentation. The manual for the Supermicro cases [1, 2] does a pretty good job IMO explaining the different options. See page D-14 and on in the 826 manual, or page D-11 and on in the 846 manual. I'll read though that, thanks for the detailed pointers. A. -- Adam Sherman CTO, Versature Corp. Tel: +1.877.498.3772 x113 ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Understanding SAS/SATA Backplanes and Connectivity
Another thought in the same vein, I notice many of these systems support SES-2 for management. Does this do anything useful under Solaris? Sorry for these questions, I seem to be having a tough time locating relevant information on the web. Thanks, A. -- Adam Sherman CTO, Versature Corp. Tel: +1.877.498.3772 x113 ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Understanding SAS/SATA Backplanes and Connectivity
We have a SC846E1 at work; it's the 24-disk, 4u version of the 826e1. It's working quite nicely as a SATA JBOD enclosure. We'll probably be buying another in the coming year to have more capacity. Good to hear. What HBA(s) are you using against it? I've got one too and it works great. I use the LSI SAS 3442e which also gives you an external SAS port. You don't need a fancy HBA with onboard RAID. Configure to IT mode. -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Understanding SAS/SATA Backplanes and Connectivity
On Thu, 16 Jul 2009 20:26:17 -0400 Adam Sherman asher...@versature.com wrote: Another thought in the same vein, I notice many of these systems support SES-2 for management. Does this do anything useful under Solaris? We've got some integration between FMA and SES devices which allows us to to some management tasks. libtopo, libscsi and libses are the main methods of getting that information out. For an example outside FMA, you could have a look into the ses/sgen plugin from pluggable fwflash. Is there anything you're specifically interested in wrt management uses of SES? thanks, James C. McPherson -- Senior Kernel Software Engineer, Solaris Sun Microsystems http://blogs.sun.com/jmcp http://www.jmcp.homeunix.com/blog Kernel Conference Australia - http://au.sun.com/sunnews/events/2009/kernel ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Understanding SAS/SATA Backplanes and Connectivity
On 16-Jul-09, at 20:52 , James C. McPherson wrote: Another thought in the same vein, I notice many of these systems support SES-2 for management. Does this do anything useful under Solaris? We've got some integration between FMA and SES devices which allows us to to some management tasks. So that would allow FMA to detect SATA disk failures then? libtopo, libscsi and libses are the main methods of getting that information out. For an example outside FMA, you could have a look into the ses/sgen plugin from pluggable fwflash. Is there anything you're specifically interested in wrt management uses of SES? I'm really just exploring. Where can I read about how FMA is going to help with failures in my setup? Thanks, A. -- Adam Sherman CTO, Versature Corp. Tel: +1.877.498.3772 x113 ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Understanding SAS/SATA Backplanes and Connectivity
On 16-Jul-09, at 18:01 , Will Murnane wrote: We have a SC846E1 at work; it's the 24-disk, 4u version of the 826e1. It's working quite nicely as a SATA JBOD enclosure. We'll probably be buying another in the coming year to have more capacity. I should also ask: any other solutions I should have a look at to get =12 SATA disks externally attached to my systems? Thanks! A. -- Adam Sherman CTO, Versature Corp. Tel: +1.877.498.3772 x113 ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Understanding SAS/SATA Backplanes and Connectivity
On Thu, 16 Jul 2009, Adam Sherman wrote: I should also ask: any other solutions I should have a look at to get =12 SATA disks externally attached to my systems? Depending on how much failure resiliancy you want and how you plan to configure your pool, you may be better off with two independent disk trays with 12 disks each. For example, if you were to use mirrors, you could split the mirrors across the disk trays. If one tray fails, then your system still works. If you are planning to use raidz1 or raidz2 then there is likely no benefit to going with two smaller trays. Bob -- Bob Friesenhahn bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/ ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Understanding SAS/SATA Backplanes and Connectivity
On Thu, Jul 16, 2009 at 20:20, Adam Shermanasher...@versature.com wrote: Ever seen/read about anyone use this kind of setup for HA clustering? I'm getting ideas about Open HA/Solaris Cluster on top of this setup with two systems connecting, that would rock! It's possible that this would work with homogeneous hardware, but I tried with another LSI-based expander and SATA disks, and had no luck. Perhaps SAS is necessary? We have a SC846E1 at work; it's the 24-disk, 4u version of the 826e1. It's working quite nicely as a SATA JBOD enclosure. We'll probably be buying another in the coming year to have more capacity. Good to hear. What HBA(s) are you using against it? LSI 3442E-R. It's connected through a Supermicro cable, CBL-0168L, so it can be attached via an external cable. There's a card needed, CSE-PTJBOD-CB1, that allows the case to run without a motherboard in it. There's no monitoring for the power supplies, but I built one for it; I can provide schematics and suggested part numbers if you're interested. We have a SC846E1 at work; it's the 24-disk, 4u version of the 826e1. It's working quite nicely as a SATA JBOD enclosure. We'll probably be buying another in the coming year to have more capacity. I should also ask: any other solutions I should have a look at to get =12 SATA disks externally attached to my systems? This was the best solution we found for the money. The 826 is about $750, while the 846 is $1100 shipped (wiredzone.com). Per disk, the 846 is almost $20 cheaper. If you only care for 12 disks, then one might as well not spend the extra money, but if there's potential for expansion it's a wise investment. Will ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Understanding SAS/SATA Backplanes and Connectivity
On Thu, Jul 16, 2009 at 21:16, Rob Loganr...@logan.com wrote: I'm confused, I though expanders only worked with SAS disk, and SATA disks took an entire SAS port. could someone post an output showing more than 4 SATA drives across one InfiniBand cable (4 SAS ports) 2 % cfgadm | grep sata sata1/0::dsk/c9t0d0 cd/dvd connected configured ok sata1/1::dsk/c9t1d0 disk connected configured ok sata1/2::dsk/c9t2d0 disk connected configured ok sata1/3 sata-port empty unconfigured ok sata1/4::dsk/c9t4d0 disk connected configured ok sata1/5 sata-port empty unconfigured ok sata2/0::dsk/c7t0d0 disk connected configured ok sata2/1::dsk/c7t1d0 disk connected configured ok sata2/2::dsk/c7t2d0 disk connected configured ok sata2/3 sata-port empty unconfigured ok sata2/4::dsk/c7t4d0 disk connected configured ok sata2/5::dsk/c7t5d0 disk connected configured ok sata2/6 sata-port empty unconfigured ok sata2/7 sata-port empty unconfigured ok sata3/0::dsk/c8t0d0 disk connected configured ok sata3/1::dsk/c8t1d0 disk connected configured ok sata3/2::dsk/c8t2d0 disk connected configured ok sata3/3 sata-port empty unconfigured ok sata3/4::dsk/c8t4d0 disk connected configured ok sata3/5::dsk/c8t5d0 disk connected configured ok sata3/6 sata-port empty unconfigured ok sata3/7 sata-port empty unconfigured ok Here's the relevant part of cfgadm -al on our machine. The disks are all sata. c4 scsi-bus connectedconfigured unknown c4::dsk/c4t15d0disk connectedconfigured unknown c4::dsk/c4t17d0disk connectedconfigured unknown c4::dsk/c4t18d0disk connectedconfigured unknown c4::dsk/c4t19d0disk connectedconfigured unknown c4::dsk/c4t20d0disk connectedconfigured unknown c4::dsk/c4t21d0disk connectedconfigured unknown c4::dsk/c4t22d0disk connectedconfigured unknown c4::dsk/c4t23d0disk connectedconfigured unknown c4::dsk/c4t24d0disk connectedconfigured unknown c4::dsk/c4t25d0disk connectedconfigured unknown c4::dsk/c4t26d0disk connectedconfigured unknown c4::dsk/c4t27d0disk connectedconfigured unknown c4::dsk/c4t28d0disk connectedconfigured unknown c4::dsk/c4t29d0disk connectedconfigured unknown c4::dsk/c4t30d0disk connectedconfigured unknown c4::dsk/c4t31d0disk connectedconfigured unknown c4::dsk/c4t32d0disk connectedconfigured unknown c4::dsk/c4t33d0disk connectedconfigured unknown c4::es/ses0ESI connectedconfigured unknown Will ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Understanding SAS/SATA Backplanes and Connectivity
c4 scsi-bus connectedconfigured unknown c4::dsk/c4t15d0disk connectedconfigured unknown : c4::dsk/c4t33d0disk connectedconfigured unknown c4::es/ses0ESI connectedconfigured unknown thanks! so SATA disks show up JBOD in IT mode.. Is there some magic that load balances the 4 SAS ports as this shows up as one scsi-bus? ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Understanding SAS/SATA Backplanes and Connectivity
On 16-Jul-09, at 21:17 , Will Murnane wrote: Good to hear. What HBA(s) are you using against it? LSI 3442E-R. It's connected through a Supermicro cable, CBL-0168L, so it can be attached via an external cable. I'm looking at the LSI SAS3801X because it seems to be what Sun OEMs for my X4100s: http://sunsolve.sun.com/handbook_private/validateUser.do?target=Devices/SCSI/SCSI_PCIX_SAS_SATA_HBA $280 or so, looks like. Might be overkill for me though. A. -- Adam Sherman CTO, Versature Corp. Tel: +1.877.498.3772 x113 ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Understanding SAS/SATA Backplanes and Connectivity
On Thu, Jul 16, 2009 at 21:35, Adam Shermanasher...@versature.com wrote: I'm looking at the LSI SAS3801X because it seems to be what Sun OEMs for my X4100s: If you're given the choice (i.e., you have the M2 revision), PCI Express is probably the bus to go with. It's basically the same card, but on a faster bus. But there's nothing wrong with the PCI-X version. http://www.lsi.com/storage_home/products_home/host_bus_adapters/sas_hbas/lsisas3801e/index.html $280 or so, looks like. Might be overkill for me though. The 3442X-R is a little cheaper: $205 from Provantage. http://www.provantage.com/lsi-logic-lsi00164~7LSIG06K.htm Will ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
