Re: Dual Pathing to SCSI/FC devices.
At 10:41 AM +1000 2000/3/31, Carl Makin wrote: > heh, the buzzword for this is "Dynamic Failover". :) In management > circles where the current focus is on 24x7, this is seen as a distinct > advantage. Okay, then I'd certainly like to have that. > "Dual Pathing" the buzzword for using both paths to the device would also > be desirable, but then you get into things like wanting to optimise data > paths depending on how busy each path is. Ahh, right. Basically you're talking about active/active load balancing. Yup, I'd like to have that, too. > No, sorry. I can't actually say what box we're buying yet since we > haven't signed the contract. :( You may be fortunate -- we were stupid enough to sign a three year lease on equipment we thought would do the job, but are now having some problems with. ;-( -- These are my opinions -- not to be taken as official Skynet policy == Brad Knowles, <[EMAIL PROTECTED]>|| Belgacom Skynet SA/NV Systems Architect, Mail/News/FTP/Proxy Admin || Rue Colonel Bourg, 124 Phone/Fax: +32-2-706.13.11/12.49 || B-1140 Brussels http://www.skynet.be || Belgium To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: Dual Pathing to SCSI/FC devices.
On Wed, 29 Mar 2000, Brad Knowles wrote: > At 4:28 PM -0800 2000/3/28, Matthew Jacob wrote: > > Yes, we very much has considered this. What's your issue about this, per se? > Myself, I just need to be able to tell the system that SCSI ID x > LUN y is actually the same logical device as SCSI ID v LUN w, but > that one is preferred and the other is backup, and have FreeBSD deal > with doing the re-targeting in the CAM SCSI driver. heh, the buzzword for this is "Dynamic Failover". :) In management circles where the current focus is on 24x7, this is seen as a distinct advantage. > The end result should be that nothing above the CAM SCSI driver > should know that a switch has occurred -- especially not programs > Same deal with fibrechannel as SCSI. > Does that about sum it up? Yes. That was pretty much what I was thinking. "Dual Pathing" the buzzword for using both paths to the device would also be desirable, but then you get into things like wanting to optimise data paths depending on how busy each path is. > Oh, and Carl -- I don't suppose you're looking at Hitachi DF400 > (sometimes rebadged as Comparex D1400) units, are you? If so, I'd No, sorry. I can't actually say what box we're buying yet since we haven't signed the contract. :( Carl. To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: Dual Pathing to SCSI/FC devices.
At 9:51 PM -0800 2000/3/28, Matthew Jacob wrote: > What's harder is *how* to use it. There are no particular hooks in FFS to > handle the multiple paths simultaneously with any coherency (for performance > reasons, should you want to do so and could prove that it'd be worthwhile), > nor are there any particular hooks that I'm aware of to do dynamic > multipathing for failover reasons at the volume or device level- if >there were > in FreeBSD, I suspect VINUM in conjunction with a completed DEVVS >might be the > place for them, but that's just random speculation on my part. My personal view is that all of this should be invisible to the higher layers, unless they know the right interfaces to use to go looking for this information. The switch should be made at the CAM SCSI layer, and applications above that don't need to know anything about it. -- These are my opinions -- not to be taken as official Skynet policy == Brad Knowles, <[EMAIL PROTECTED]>|| Belgacom Skynet SA/NV Systems Architect, Mail/News/FTP/Proxy Admin || Rue Colonel Bourg, 124 Phone/Fax: +32-2-706.13.11/12.49 || B-1140 Brussels http://www.skynet.be || Belgium To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: Dual Pathing to SCSI/FC devices.
At 3:24 PM +1000 2000/3/29, Carl Makin wrote: > There doesn't seem much point in using VINUM with an external RAID5 box > (although we use VINUM a lot on other machines). Speaking only for myself and the Comparex D1400 (Hitachi DF400) unit that I've been playing with so far, there is *very* good reason to use vinum on top of the external drive array. In particular, the unit has five SCSI busses internally, and when you configure LUNs you typically configure one or more rows of five disks per row. However, when you configure more than one row for a LUN, it simply appends one row to the next, and doesn't stripe or RAID-5 across all disks simultaneously. This really kills your performance. If I instead create one LUN for each row of disks (RAID-5), I can then stripe across them in software using vinum, and start approaching the levels of performance that I could get if I instead were using vinum exclusively and striping across twenty high-speed disks on five separate SCSI busses that were directly connected to the host. Unfortunately, this device doesn't have any concept of creating LUNs composed of other LUNs, so that I could keep all this bizarreness strictly within the cabinet. > The box to which I'm hooking this up has sufficient performance to handle > 16 U2W SCSI links running hard. Being able to utilise multiple paths > could be a big performance win. Theoretically, the box I'm connecting to can handle that level of performance, too. However, I've had problems with it when I push it hard with just two Adaptec 2940U2W host adaptors, each connecting to one interface on separate device controllers. In theory, practice and theory are the same. In practice, they are frequently radically different. > I'm getting a little beyond my depth in CAM internals here. But it is > *very* interesting. :) I'm out of my depth in CAM SCSI internals, too but I'm also willing to provide whatever assistance I can, with regards to testing. -- These are my opinions -- not to be taken as official Skynet policy == Brad Knowles, <[EMAIL PROTECTED]>|| Belgacom Skynet SA/NV Systems Architect, Mail/News/FTP/Proxy Admin || Rue Colonel Bourg, 124 Phone/Fax: +32-2-706.13.11/12.49 || B-1140 Brussels http://www.skynet.be || Belgium To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: Dual Pathing to SCSI/FC devices.
At 4:28 PM -0800 2000/3/28, Matthew Jacob wrote: > Yes, we very much has considered this. What's your issue about this, per se? > Right now there's no framework code to directly exploit or prohibit multiple > paths to the same disk, whether via Fibre Channel or SCSI. Myself, I just need to be able to tell the system that SCSI ID x LUN y is actually the same logical device as SCSI ID v LUN w, but that one is preferred and the other is backup, and have FreeBSD deal with doing the re-targeting in the CAM SCSI driver. The end result should be that nothing above the CAM SCSI driver should know that a switch has occurred -- especially not programs like vinum, which might be called upon to do software mirroring/striping/RAID on top of the hardware mirroring/striping/RAID. Same deal with fibrechannel as SCSI. Does that about sum it up? Oh, and Carl -- I don't suppose you're looking at Hitachi DF400 (sometimes rebadged as Comparex D1400) units, are you? If so, I'd like to talk to you in some depth about these things -- maybe you can come up with some solutions for some problems I've had. -- These are my opinions -- not to be taken as official Skynet policy == Brad Knowles, <[EMAIL PROTECTED]>|| Belgacom Skynet SA/NV Systems Architect, Mail/News/FTP/Proxy Admin || Rue Colonel Bourg, 124 Phone/Fax: +32-2-706.13.11/12.49 || B-1140 Brussels http://www.skynet.be || Belgium To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: Dual Pathing to SCSI/FC devices.
> > On Tue, 28 Mar 2000, Matthew Jacob wrote: > > > Umm, okay. Is this with Veritas DMP or VCS? > > Good question. DMP I think. I'm not sure what VCS is. Veritas Cluster Server > > > an active-active configuration. If you use Greg Lehey's VINUM in FreeBSD, I'm > > not clear about what it's role in terms of recognizing redundant paths might > > There doesn't seem much point in using VINUM with an external RAID5 box > (although we use VINUM a lot on other machines). Well, you get volume (plex) identification out of it, which is of help when you don't want to try and keep track of where disks end up over time manually. > > identification restrictions that could cause an upset. The CAM midlayer does > > track Vital Product Data (like drive serial numbers), but this is put together > > with a bus address (e.g, HBA, bus on HBA, target, lun) to track whether > > particular device has changed at that location or not- not whether that > > particular spindle is replicated elsewhere. > > Would it be hard to add intelligence to this layer that would detect if a > drive (via it's VPD info) is visible on multiple busses? If so is there a > point in CAM where it could be modified to handle, at the minimum, a > redundant path or better yet, use multiple paths to a device? > > The box to which I'm hooking this up has sufficient performance to handle > 16 U2W SCSI links running hard. Being able to utilise multiple paths > could be a big performance win. You can indeed add intelligence, but it's a very wide open question how and where the intelligence should be used. And the "why" is of importance as well. You essentially need to coordiante object and path identification with volume and filesystem usage. Right now there's no problem about having multiple paths to the same disk- the only piece that's really missing is some moderately easy method to export this information out of any specific layer- but this isn't that hard to do, and could, in fact, be done via a user daemon without any kernel modifications, so I don't view this as a hard problem. What's harder is *how* to use it. There are no particular hooks in FFS to handle the multiple paths simultaneously with any coherency (for performance reasons, should you want to do so and could prove that it'd be worthwhile), nor are there any particular hooks that I'm aware of to do dynamic multipathing for failover reasons at the volume or device level- if there were in FreeBSD, I suspect VINUM in conjunction with a completed DEVVS might be the place for them, but that's just random speculation on my part. > > > Absolutely. I mean, we do have a supported Fibre Channel card (Qlogic 2100 and > > Qlogic 2200), but not as much testing as one would like. > > I have a Qlogic 2200 on order to test this out. Good! You'll find that this is better with respect to full fabric support if you happen to have, e.g., a Brocade switch on order. > > > You should note that there isn't, for fibre channel, any particular address > > wiring enforced except by that which the target device does (e.g., if the > > Would this be similar to the situation that we had with FreeBSD before you > could wire down devices? Yes, somewhat, but not quite as bad. > > > Let me know if this is enough info to help. > > I'm getting a little beyond my depth in CAM internals here. But it is > *very* interesting. :) > > Thanks, You're welcome. Lemme know how it goes. -matt To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: Dual Pathing to SCSI/FC devices.
On Tue, 28 Mar 2000, Matthew Jacob wrote: > Umm, okay. Is this with Veritas DMP or VCS? Good question. DMP I think. I'm not sure what VCS is. > an active-active configuration. If you use Greg Lehey's VINUM in FreeBSD, I'm > not clear about what it's role in terms of recognizing redundant paths might There doesn't seem much point in using VINUM with an external RAID5 box (although we use VINUM a lot on other machines). > identification restrictions that could cause an upset. The CAM midlayer does > track Vital Product Data (like drive serial numbers), but this is put together > with a bus address (e.g, HBA, bus on HBA, target, lun) to track whether > particular device has changed at that location or not- not whether that > particular spindle is replicated elsewhere. Would it be hard to add intelligence to this layer that would detect if a drive (via it's VPD info) is visible on multiple busses? If so is there a point in CAM where it could be modified to handle, at the minimum, a redundant path or better yet, use multiple paths to a device? The box to which I'm hooking this up has sufficient performance to handle 16 U2W SCSI links running hard. Being able to utilise multiple paths could be a big performance win. > Absolutely. I mean, we do have a supported Fibre Channel card (Qlogic 2100 and > Qlogic 2200), but not as much testing as one would like. I have a Qlogic 2200 on order to test this out. > You should note that there isn't, for fibre channel, any particular address > wiring enforced except by that which the target device does (e.g., if the Would this be similar to the situation that we had with FreeBSD before you could wire down devices? > Let me know if this is enough info to help. I'm getting a little beyond my depth in CAM internals here. But it is *very* interesting. :) Thanks, Carl. To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: Dual Pathing to SCSI/FC devices.
> On Tue, 28 Mar 2000, Matthew Jacob wrote: > > > Yes, we very much has considered this. What's your issue about this, per se? > > Well, the driver for asking was my management asking if FreeBSD supported > this, as we're going to do it on AIX (with the Dual Pathing Option) and > with Solaris (running Veritas which I need to investigate more). We're > reasonably big on redundancy here and we're running a number of critical > systems on FreeBSD (DNS, WINS, DHCP, primary webserver, PDF server, etc). Umm, okay. Is this with Veritas DMP or VCS? > > > Right now there's no framework code to directly exploit or prohibit multiple > > paths to the same disk, whether via Fibre Channel or SCSI. > > Ok, but I suspect it would get a trifle upset if I started duplicating > LUNS. :) I really don't know how much work this would be but would be > interested in helping. No, it wouldn't get upset, but it wouldn't know to not get upset either :-). Formally speaking, the current non-cognizant arrangement of FreeBSD would be an active-active configuration. If you use Greg Lehey's VINUM in FreeBSD, I'm not clear about what it's role in terms of recognizing redundant paths might be, but other than that there are no particular volume or spindle identification restrictions that could cause an upset. The CAM midlayer does track Vital Product Data (like drive serial numbers), but this is put together with a bus address (e.g, HBA, bus on HBA, target, lun) to track whether particular device has changed at that location or not- not whether that particular spindle is replicated elsewhere. > > I'm not a kernel hacker but I can offer testing resources (well, in about > a month or so when the box arrives) for SCSI and possibly Fibre Channel if > someone has code. Absolutely. I mean, we do have a supported Fibre Channel card (Qlogic 2100 and Qlogic 2200), but not as much testing as one would like. You should note that there isn't, for fibre channel, any particular address wiring enforced except by that which the target device does (e.g., if the target does hard loop addressing). I've been mulling over some persistent device attribute stuff for the next round of changes (e.g., binding a particular WWN to a particular 'target'), probably stored in card NVRAM, and I've had on my 'futures' list a 'WWN' extension to devfs, but this is all futures. Let me know if this is enough info to help. -matt To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: Dual Pathing to SCSI/FC devices.
Hi Matthew, On Tue, 28 Mar 2000, Matthew Jacob wrote: > Yes, we very much has considered this. What's your issue about this, per se? Well, the driver for asking was my management asking if FreeBSD supported this, as we're going to do it on AIX (with the Dual Pathing Option) and with Solaris (running Veritas which I need to investigate more). We're reasonably big on redundancy here and we're running a number of critical systems on FreeBSD (DNS, WINS, DHCP, primary webserver, PDF server, etc). > Right now there's no framework code to directly exploit or prohibit multiple > paths to the same disk, whether via Fibre Channel or SCSI. Ok, but I suspect it would get a trifle upset if I started duplicating LUNS. :) I really don't know how much work this would be but would be interested in helping. I'm not a kernel hacker but I can offer testing resources (well, in about a month or so when the box arrives) for SCSI and possibly Fibre Channel if someone has code. Carl. To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: Dual Pathing to SCSI/FC devices.
Yes, we very much has considered this. What's your issue about this, per se? Right now there's no framework code to directly exploit or prohibit multiple paths to the same disk, whether via Fibre Channel or SCSI. On Wed, 29 Mar 2000, Carl Makin wrote: > > Is anyone working on/considering this? > > I'm about to start hooking FreeBSD boxes up to a *big* disk array which > has the ability to make LUNS appear on multiple interfaces. Being able to > access LUNS via multiple paths could be a reasonable performance gain, as > well as enhancing reliability. > > > Carl. > > > > > > To Unsubscribe: send mail to [EMAIL PROTECTED] > with "unsubscribe freebsd-current" in the body of the message > To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Dual Pathing to SCSI/FC devices.
Is anyone working on/considering this? I'm about to start hooking FreeBSD boxes up to a *big* disk array which has the ability to make LUNS appear on multiple interfaces. Being able to access LUNS via multiple paths could be a reasonable performance gain, as well as enhancing reliability. Carl. To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
