Re: Dynamic PAV Device Limit?
In a message dated 9/9/2007 8:34:21 P.M. Central Daylight Time, [EMAIL PROTECTED] writes: >I think the discussion may be accidentally creating confusion about OE and channel limitations at a single device level. The number of OE on a channel is the sum of all IO on the channel no matter how many volumes and LCU are sharing it. I understand this point. I was using only one device in my discussion as the extreme case, where extreme means from the point of view of how it used to work, namely only one I/O at a time to any given device. Bill Fairchild Plainfield, IL ** See what's new at http://www.aol.com -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: Dynamic PAV Device Limit?
Bill > > Based on everything I have read lately, I now believe it is possible to > start 256 I/O requests from one LPAR to the > same device before the first one ends if that device is in an ESS LCU > with > four FICON channels between that > LCU and the LPAR that does the I/Os, if that LPAR has the device > defined as > a PAV base device with 255 alias > devices in the same LCU, and if all four FICON channels are in the > base > device's UCB channel path matrix. That's only if you are connected to a z9. The z990, z900 and G6 have less OE per channel so you need 8 channels to get 256 IO. There can be no other IO to any other LCU daisy chained on the same channel(s). The OE supported by the storage port must also not be exhausted due to fan-in of multiple channels. In a simple model of 4xchannels dedicated to one LCU with one base and 255 alias you are correct. > These I/Os will appear to be simultaneous, but at any given instant, > no more > than 64 of the 256 will be actively > transferring data to or from the LCU. The other 192 will be > disconnected > for various reasons. Not true. There is no disconnect on FICON. The channel is moving frames or not. The number of OE with data or command in a frame on the channel will depend on how the particular server and storage models handle interleaving. This is a theoretical > maximum and not necessarily a practical limit. If eight ESCON > channels are > the link between the device and the > ESS LCU, then the number of simultaneously startable I/Os to the device > will > be more than eight but smaller than 256. In tests with zero cache hits I've seen channels with average 47 OE - an average meaning there are times when there were larger and smaller numbers. That's with 1024 spindles and 64x4Gb channels. > > Is all of this correct? > > I have been away from the bleeding edge too long. :-( I think the discussion may be accidentally creating confusion about OE and channel limitations at a single device level. The number of OE on a channel is the sum of all IO on the channel no matter how many volumes and LCU are sharing it. Ron -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: Dynamic PAV Device Limit?
In a message dated 9/7/2007 10:03:18 P.M. Central Daylight Time, [EMAIL PROTECTED] writes: >I suggest browsing Pat Artis' website for some quality education on FICON. I am doing that now and browsing the general Internet, too. I have learned a lot over the years from Pat Artis. Based on everything I have read lately, I now believe it is possible to start 256 I/O requests from one LPAR to the same device before the first one ends if that device is in an ESS LCU with four FICON channels between that LCU and the LPAR that does the I/Os, if that LPAR has the device defined as a PAV base device with 255 alias devices in the same LCU, and if all four FICON channels are in the base device's UCB channel path matrix. These I/Os will appear to be simultaneous, but at any given instant, no more than 64 of the 256 will be actively transferring data to or from the LCU. The other 192 will be disconnected for various reasons. This is a theoretical maximum and not necessarily a practical limit. If eight ESCON channels are the link between the device and the ESS LCU, then the number of simultaneously startable I/Os to the device will be more than eight but smaller than 256. Is all of this correct? I have been away from the bleeding edge too long. :-( Bill Fairchild Plainfield, IL ** See what's new at http://www.aol.com -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: Dynamic PAV Device Limit?
Ted > > But, in general, I think the law of diminishing returns kicks in at > around 6-8. > Maybe true when PG are only using 4-8 spindles, but with wide striping and large volumes a poor cache hit workloads would get a lot of benefit from 32 spindles concurrently handling IO for a volume. Ron -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: Dynamic PAV Device Limit?
Bill, > > I confess to ignorance of FICON details and of the acronym OE. > OE is short for Open Exchanges. This refers to the number of Initiator => Target relationships supported by the Host Channel and the Storage Port. If volume has 4xFICON channels and 256 Alias then technically there is no reason why there cannot be 256 IO outstanding to that volume. > > IOS uses a PAV alias UCB to find the base UCB that is associated with > that > alias device number. > This base UCB has in it a matrix of up to eight CHPIDs that can be used > for > an I/O from that LPAR to that device. > There is only room in the UCB for eight CHPIDs. I used to think that > the > maximum number of I/Os that can be started > on any one LPAR to one particular device was eight, and that was > because I > assumed the Channel > Subsystem would not accept more than eight. I forgot about the > multiplexing > of multiple I/Os down > one channel. After reading your post and rereading the PoOps section > on the > I/O Start Function, it appears that > if the Channel Subsystem finds all eight (maximum possible paths to a > device) channels busy (connected), then > that I/O is queued in the CS; i.e., it remains Start Function pending. > The > CS must store this request in a queue > somewhere in its HSA control blocks, and later the CS will start the > 9th > through the n-th I/O to the same device > down whichever channel is next to indicate that it is disconnected. > No > matter how many "quasi-simultaneous" I/Os > that the CS has started to the device, however, at any one instant no > more > than eight of them can possibly be connected > to a channel path and actively transferring data. True, but once you are in IOS you are already past the UCB. That's why it is called IOS Queue :-) With ESCON there can up to eight channels on an LCU transferring data, and an even greater number of IO Disconnected while a volume's disk is seeking, queued for sibling pend, waiting for remote copy IO, waiting for dirty cache delays, delayed for XRC sidefile full, etc. > > The disconnect-reconnect capability has been present since the early > 1970s > with Block Multiplexor > Channels. Whenever a control unit sees a command in the channel > program > that could result in a > significant amount of time before that command can finish (such as a > seek), > the control unit disconnects > that I/O operation from the channel to which is was connected. Then > that > channel is available to do I/O > to or from any device that is accessible through that channel. I used > to > think that only an I/O to a > different device would steal away the channel at this point. But with > the > advent of PAV it now seems > that a different I/O to the same device could connect to the channel. > Or > maybe that possibility has been in the > microcode since even before PAV. > And of none of the preceeding discussion (mine and yours) applies to FICON. FICON does not really have a connect and disconnect state. It is more like transferring and not transferring frames for a volume. There is no DPR - everything happens on the same channel. Think of it like a PABX that supports 64 phone calls. When 64 lines are in use you have to wait. It happens rarely for good cache IO, but for some cache unfriendly workloads (CA-IDMS and IMS Fastpath spring to mind) you can quickly use up the OE and watch IO stall waiting for cache misses to complete. > > The patent applications to which I referred in my earlier post were > those of > EMC's initial support of > 2105 PAV compatibility in late 2000. There is a limit of eight per > device > that was architected into that level of > microcode. Evidently they have increased the limit to some new, much > larger > value. Some vendors wrote their own code to talk to PAV on MVS, and some paid IBM for the API and still had limits initially. The limits on the storage were not PAV limits in MVS. Note that there are other resource restrictions that can affect FICON throughput. I suggest browsing Pat Artis' website for some quality education on FICON. > > Bill Fairchild > Plainfield, IL > > > > > -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: Dynamic PAV Device Limit?
>I have not been able to find any statement about whether WLM will put any >limit on the number of aliases that can be assigned to a particular device >when using Dynamic PAVs. IIRC, it's 256, but I have seen other vendors limit in the hardware (EMC). The highest I have ever seen is 50. And, that was solving a problem we didn't know we had. An IMS application was writing debugging information to SYSOUT (yes, I know). Due to the number of MPP's, one SPOOL volume was getting pounded. But, in general, I think the law of diminishing returns kicks in at around 6-8. - Too busy driving to stop for gas! -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: Dynamic PAV Device Limit?
In a message dated 9/7/2007 11:47:40 A.M. Central Daylight Time, [EMAIL PROTECTED] writes: >This is not quite right. FICON is restricted by the either the number of OE supported by the channel, or the OE supported by the Storage port. On a z9 there can be 64 OE on every channel on an LCU - with eight channels that's 512 IO that can be transferring data or disconnected and waiting for a cache miss, TrueCopy RIO or such like at the same time. I confess to ignorance of FICON details and of the acronym OE. >AFAIK there is no MVS restriction on the number of alias assigned to a device. You can have one base address with all 255 alias assigned if that is something useful. Datasets that are good candidate for FlashAccess would probably benefit for this sort of unique setup. High activity volumes with a good cache hit rate on Wide Stripe PG would also benefit. There were some restriction on the number of alias in EMC and HDS a few generations ago, but not in current models. IOS uses a PAV alias UCB to find the base UCB that is associated with that alias device number. This base UCB has in it a matrix of up to eight CHPIDs that can be used for an I/O from that LPAR to that device. There is only room in the UCB for eight CHPIDs. I used to think that the maximum number of I/Os that can be started on any one LPAR to one particular device was eight, and that was because I assumed the Channel Subsystem would not accept more than eight. I forgot about the multiplexing of multiple I/Os down one channel. After reading your post and rereading the PoOps section on the I/O Start Function, it appears that if the Channel Subsystem finds all eight (maximum possible paths to a device) channels busy (connected), then that I/O is queued in the CS; i.e., it remains Start Function pending. The CS must store this request in a queue somewhere in its HSA control blocks, and later the CS will start the 9th through the n-th I/O to the same device down whichever channel is next to indicate that it is disconnected. No matter how many "quasi-simultaneous" I/Os that the CS has started to the device, however, at any one instant no more than eight of them can possibly be connected to a channel path and actively transferring data. >Note that ESCON also did not have a restriction of one IO per channel. There could only be one data transfer per ESCON channel, but when an IO disconnected another IO could jump straight in and use the channel. There could be any number of IO in a Disconnect state, but only one IO per channel connected. The disconnect-reconnect capability has been present since the early 1970s with Block Multiplexor Channels. Whenever a control unit sees a command in the channel program that could result in a significant amount of time before that command can finish (such as a seek), the control unit disconnects that I/O operation from the channel to which is was connected. Then that channel is available to do I/O to or from any device that is accessible through that channel. I used to think that only an I/O to a different device would steal away the channel at this point. But with the advent of PAV it now seems that a different I/O to the same device could connect to the channel. Or maybe that possibility has been in the microcode since even before PAV. The patent applications to which I referred in my earlier post were those of EMC's initial support of 2105 PAV compatibility in late 2000. There is a limit of eight per device that was architected into that level of microcode. Evidently they have increased the limit to some new, much larger value. Bill Fairchild Plainfield, IL ** Get a sneak peek of the all-new AOL at http://discover.aol.com/memed/aolcom30tour -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: Dynamic PAV Device Limit?
Bill, This is not quite right. FICON is restricted by the either the number of OE supported by the channel, or the OE supported by the Storage port. On a z9 there can be 64 OE on every channel on an LCU - with eight channels that's 512 IO that can be transferring data or disconnected and waiting for a cache miss, TrueCopy RIO or such like at the same time. AFAIK there is no MVS restriction on the number of alias assigned to a device. You can have one base address with all 255 alias assigned if that is something useful. Datasets that are good candidate for FlashAccess would probably benefit for this sort of unique setup. High activity volumes with a good cache hit rate on Wide Stripe PG would also benefit. There were some restriction on the number of alias in EMC and HDS a few generations ago, but not in current models. Note that ESCON also did not have a restriction of one IO per channel. There could only be one data transfer per ESCON channel, but when an IO disconnected another IO could jump straight in and use the channel. There could be any number of IO in a Disconnect state, but only one IO per channel connected. Ron > -Original Message- > From: IBM Mainframe Discussion List [mailto:[EMAIL PROTECTED] On > Behalf Of (IBM Mainframe Discussion List) > Sent: Friday, September 07, 2007 6:40 AM > To: IBM-MAIN@BAMA.UA.EDU > Subject: Re: [IBM-MAIN] Dynamic PAV Device Limit? > > > > In a message dated 9/7/2007 7:59:47 A.M. Central Daylight Time, > [EMAIL PROTECTED] writes: > >I have not been able to find any statement about whether WLM will put > any limit on the number of aliases that can be assigned to a > particular device when using Dynamic PAVs. Does anyone on the list > know of a limit? Can anyone point me to a manual that would document > whether this can occur or not? > > I don't know what WLM uses as a limit, if any, but the I/O hardware has > a > limit of eight simultaneously active I/O > requests to any one given device. Even if you have 100 aliases, there > can > still only be eight I/Os involving that one > device occurring at the same time. These eight can come from eight > different LPARs or all from one LPAR. This hardware > limit of eight is true whether the PAVs are dynamic or static. I > can't > quickly find a document with the number eight in it, > but it's in patent applications that I have studied. > > Bill Fairchild > Plainfield, IL > > > > > > ** Get a sneak peek of the all-new > AOL at > http://discover.aol.com/memed/aolcom30tour > > -- > For IBM-MAIN subscribe / signoff / archive access instructions, > send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO > Search the archives at http://bama.ua.edu/archives/ibm-main.html -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: Dynamic PAV Device Limit?
On our IBM SHARKS F2105's and 8100's; we have 48 PAV's defined to each LCU. I have seen as many as 47 PAVs defined to 1 logical device. Water is put where ever the FIRE IS with Dynamic PAV's. The restriction is the number you have GEN'ed to the LCU. I do not think other DASD Vendors do this. I know at one point in time EMC has a maximum of 8 PAV's per logical device and do not know if that number has been increased. For example In your I/O GEN Real device are gen'ed as 3390B for bases PAV devices are gen'ed as 3390A for Aliases IODEVICE ADDRESS=(4000,208),CUNUMBR=(AF40),STADET=Y,UNIT=3390B IODEVICE ADDRESS=(40D0,048),CUNUMBR=(AF40),STADET=Y,UNIT=3390A Total 256 devices per LCU. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: Dynamic PAV Device Limit?
In a message dated 9/7/2007 7:59:47 A.M. Central Daylight Time, [EMAIL PROTECTED] writes: >I have not been able to find any statement about whether WLM will put any limit on the number of aliases that can be assigned to a particular device when using Dynamic PAVs. Does anyone on the list know of a limit? Can anyone point me to a manual that would document whether this can occur or not? I don't know what WLM uses as a limit, if any, but the I/O hardware has a limit of eight simultaneously active I/O requests to any one given device. Even if you have 100 aliases, there can still only be eight I/Os involving that one device occurring at the same time. These eight can come from eight different LPARs or all from one LPAR. This hardware limit of eight is true whether the PAVs are dynamic or static. I can't quickly find a document with the number eight in it, but it's in patent applications that I have studied. Bill Fairchild Plainfield, IL ** Get a sneak peek of the all-new AOL at http://discover.aol.com/memed/aolcom30tour -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: Dynamic PAV Device Limit?
. I have not been able to find any statement about whether WLM will put any limit on the number of aliases that can be assigned to a particular device when using Dynamic PAVs. Does anyone on the list know of a limit? Can anyone point me to a manual that would document whether this can occur or not? . Tom, no limits on Alias associated to a Base address by WLM. If your DASD IOSQ time is high, WLM can add Aliases to Base address until IOSQ time goes down. Sometimes WLM limits the number of Aliases if it determines that adding Aliases could interfere with other response times components, like DISCONNECT times or so on. In this case WLM has an arbitrary limit. Hope this helps. _ Paolo Cacciari Business Continuity and Resiliency Services, IBM Global Services - South Region, EMEA Via Darwin 85, 20019 Settimo Milanese(MI) – Italy - MISET001 "The goal is to be prepared for a disaster not to continually plan for a successful test" * [EMAIL PROTECTED] ( + 39 051 41.36799 Mobile: + 39 335 6287584 7 + 39 02 596.23288 Fax BO: + 39 051 406052 -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: Dynamic PAV Device Limit?
I would look in the storage subsystem documentation. Regards, Jihad K. Kawkabani Voice: 440.395.0740 Network: 575.0740 Cell: 440.465.2969 Tom Moulder <[EMAIL PROTECTED] M.NET> To Sent by: IBM IBM-MAIN@BAMA.UA.EDU Mainframe cc Discussion List <[EMAIL PROTECTED] Subject .EDU> Dynamic PAV Device Limit? 09/07/2007 08:59 AM Please respond to IBM Mainframe Discussion List <[EMAIL PROTECTED] .EDU> I have not been able to find any statement about whether WLM will put any limit on the number of aliases that can be assigned to a particular device when using Dynamic PAVs. Does anyone on the list know of a limit? Can anyone point me to a manual that would document whether this can occur or not? Tom Moulder -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Dynamic PAV Device Limit?
I have not been able to find any statement about whether WLM will put any limit on the number of aliases that can be assigned to a particular device when using Dynamic PAVs. Does anyone on the list know of a limit? Can anyone point me to a manual that would document whether this can occur or not? Tom Moulder -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
