Re: Any ROT for DASD Response time
also, as some very wise people on this list and other lists/papers have pointed out Know thy data some of our worst dasd response time issues were eliminated by simply looking at the logical volume placement and the data sets there on. Moving some DB2 indexes into different array groups and/or off logical volumes that were splattered out towards the middle of the small spinning back end dasd can make a great deal of difference (know your chunk sizes, how the chunks are spread across the disks in the arrays, etc...). RAID technology (at least yet) doesn't make up for bad placement of storage group volumes and the data sets on them. As others pointed out, research Dr. Artis papers on the subject--just my opinion and experience. (Because we've always done it this way *BANG*)---Running HSM migration management/space release-along with compactor jobs on different lpar in the plex- on logical volumes that were on the front of the disks in the arrays absolutely killed the hot indexes that were in the middle of the disks in the same array. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: Any ROT for DASD Response time
J, This is what Pat describes as SIBLING PEND and it is easy to spot the symptom because it shows up as Disconnect Time. I sure IBM and EMC have something like HDS's Performance Monitor. It's a GUI with exportable data that you can use to monitor skewed activity across the parity groups that lead to the elongated seek affect that you describe. Ron -Original Message- From: IBM Mainframe Discussion List [mailto:ibm-m...@bama.ua.edu] On Behalf Of J Ellis Sent: Thursday, July 29, 2010 4:47 AM To: IBM-MAIN@bama.ua.edu Subject: Re: [IBM-MAIN] Any ROT for DASD Response time also, as some very wise people on this list and other lists/papers have pointed out Know thy data some of our worst dasd response time issues were eliminated by simply looking at the logical volume placement and the data sets there on. Moving some DB2 indexes into different array groups and/or off logical volumes that were splattered out towards the middle of the small spinning back end dasd can make a great deal of difference (know your chunk sizes, how the chunks are spread across the disks in the arrays, etc...). RAID technology (at least yet) doesn't make up for bad placement of storage group volumes and the data sets on them. As others pointed out, research Dr. Artis papers on the subject--just my opinion and experience. (Because we've always done it this way *BANG*)---Running HSM migration management/space release-along with compactor jobs on different lpar in the plex- on logical volumes that were on the front of the disks in the arrays absolutely killed the hot indexes that were in the middle of the disks in the same array. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: Any ROT for DASD Response time
The first thing you have to determine is the breakout of the DASD RT. What is the RT and how much of it is due to connect vs. disc vs. IOSQ vs. pend. From there you can start understanding if you have an I/O RT problem and what might be causing it. Joel Wolpert Performance and Capacity Planning consultant WEBSITE: www.perfconsultant.com - Original Message - From: Lizette Koehler stars...@mindspring.com Newsgroups: bit.listserv.ibm-main To: IBM-MAIN@bama.ua.edu Sent: Wednesday, July 21, 2010 12:39 PM Subject: Any ROT for DASD Response time We are running an EMC DMX4500 shared with the open system side. MVS has 9TB and the open side has 500TB. So they win. We have points in time where our production application (DB2 Stored Procedures with Native SQL) has slow responses. The DBAs are using RMF and TMON (DB2 and MVS) to isolate the problem. However, they see DFHSM and say thats the problem. I think I am seeing delays on the dasd the DB2 system uses in RMF. But I am not sure how to interpret the numbers. Is there a ROT that says if the delay rate is ? then the dasd is poor performaing? Thanks Lizette -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: Any ROT for DASD Response time
Disconnect time can also show up as synchronous remote copy (SRDF). If the links are saturated it will elongate the DISC time and therefore the RT. Joel Wolpert Performance and Capacity Planning consultant WEBSITE: www.perfconsultant.com - Original Message - From: Ron Hawkins ron.hawkins1...@sbcglobal.net Newsgroups: bit.listserv.ibm-main To: IBM-MAIN@bama.ua.edu Sent: Thursday, July 29, 2010 11:59 AM Subject: Re: Any ROT for DASD Response time J, This is what Pat describes as SIBLING PEND and it is easy to spot the symptom because it shows up as Disconnect Time. I sure IBM and EMC have something like HDS's Performance Monitor. It's a GUI with exportable data that you can use to monitor skewed activity across the parity groups that lead to the elongated seek affect that you describe. Ron -Original Message- From: IBM Mainframe Discussion List [mailto:ibm-m...@bama.ua.edu] On Behalf Of J Ellis Sent: Thursday, July 29, 2010 4:47 AM To: IBM-MAIN@bama.ua.edu Subject: Re: [IBM-MAIN] Any ROT for DASD Response time also, as some very wise people on this list and other lists/papers have pointed out Know thy data some of our worst dasd response time issues were eliminated by simply looking at the logical volume placement and the data sets there on. Moving some DB2 indexes into different array groups and/or off logical volumes that were splattered out towards the middle of the small spinning back end dasd can make a great deal of difference (know your chunk sizes, how the chunks are spread across the disks in the arrays, etc...). RAID technology (at least yet) doesn't make up for bad placement of storage group volumes and the data sets on them. As others pointed out, research Dr. Artis papers on the subject--just my opinion and experience. (Because we've always done it this way *BANG*)---Running HSM migration management/space release-along with compactor jobs on different lpar in the plex- on logical volumes that were on the front of the disks in the arrays absolutely killed the hot indexes that were in the middle of the disks in the same array. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: Any ROT for DASD Response time
Joel, That's when you start looking at the Type42 subtype 6 records. Read disconnect time is reported separately, so you can establish if the problem is TrueCopy links or Sibling Pend. Ron -Original Message- From: IBM Mainframe Discussion List [mailto:ibm-m...@bama.ua.edu] On Behalf Of Joel Wolpert Sent: Thursday, July 29, 2010 2:37 PM To: IBM-MAIN@bama.ua.edu Subject: Re: [IBM-MAIN] Any ROT for DASD Response time Disconnect time can also show up as synchronous remote copy (SRDF). If the links are saturated it will elongate the DISC time and therefore the RT. Joel Wolpert Performance and Capacity Planning consultant WEBSITE: www.perfconsultant.com - Original Message - From: Ron Hawkins ron.hawkins1...@sbcglobal.net Newsgroups: bit.listserv.ibm-main To: IBM-MAIN@bama.ua.edu Sent: Thursday, July 29, 2010 11:59 AM Subject: Re: Any ROT for DASD Response time J, This is what Pat describes as SIBLING PEND and it is easy to spot the symptom because it shows up as Disconnect Time. I sure IBM and EMC have something like HDS's Performance Monitor. It's a GUI with exportable data that you can use to monitor skewed activity across the parity groups that lead to the elongated seek affect that you describe. Ron -Original Message- From: IBM Mainframe Discussion List [mailto:ibm-m...@bama.ua.edu] On Behalf Of J Ellis Sent: Thursday, July 29, 2010 4:47 AM To: IBM-MAIN@bama.ua.edu Subject: Re: [IBM-MAIN] Any ROT for DASD Response time also, as some very wise people on this list and other lists/papers have pointed out Know thy data some of our worst dasd response time issues were eliminated by simply looking at the logical volume placement and the data sets there on. Moving some DB2 indexes into different array groups and/or off logical volumes that were splattered out towards the middle of the small spinning back end dasd can make a great deal of difference (know your chunk sizes, how the chunks are spread across the disks in the arrays, etc...). RAID technology (at least yet) doesn't make up for bad placement of storage group volumes and the data sets on them. As others pointed out, research Dr. Artis papers on the subject--just my opinion and experience. (Because we've always done it this way *BANG*)---Running HSM migration management/space release-along with compactor jobs on different lpar in the plex- on logical volumes that were on the front of the disks in the arrays absolutely killed the hot indexes that were in the middle of the disks in the same array. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: Any ROT for DASD Response time
Hal, OK, the OP is believing how RMF III reports DASD Delays (not contention). Let's stop at that basic error. Connect time is counted as Using in RMF III, so opportunities to reduce IO elapsed time by reducing IO count, as you described, will never be found with RMF III delay analysis unless there is some substantial Disconnect or Pending Time associated with the total IO count. You have to look at the USING value as well, and with current Disk subsystems I would suggest eliminating that as the potential tuning opportunity (technically or politically) before moving to the RMF III delay metrics. Remember RMF III still thinks we have real 3990s. Ron -Original Message- From: IBM Mainframe Discussion List [mailto:ibm-m...@bama.ua.edu] On Behalf Of Hal Merritt Sent: Tuesday, July 27, 2010 8:14 AM To: IBM-MAIN@bama.ua.edu Subject: Re: [IBM-MAIN] Any ROT for DASD Response time See imbedded. Hal -Original Message- From: IBM Mainframe Discussion List [mailto:ibm-m...@bama.ua.edu] On Behalf Of Ron Hawkins Sent: Thursday, July 22, 2010 10:54 PM To: IBM-MAIN@bama.ua.edu Subject: Re: Any ROT for DASD Response time Hal, I should have been clear. The statement I disagreed with is: It does not really matter what the response times are, the issue is that they are too slow I'm not sure where I can agree with you, and I still fail to see how near perfect response time can be too slow. If the DASD is delivering near perfect response times, then it can't go faster. If that's still not fast enough, then one -must- address I/O reduction. War story: most loved online was hitting a ceiling and just wouldn't scale past a specific transaction load. The cause was traced to a single checkpoint dataset. The fastest hardware available was brought in but it was not fast enough. The very reluctant, politically powerful support team was forced to address the I/O to that dataset. A relatively minor change solved the problem. The point is that making application changes may meet with a -lot- of resistance, but ya gotta do whatcha gotta do. To be fair, -any- change to that application was a medium to high risk. Any volume averaging 0.3ms response time is pretty close to zero delay, where a delay is something queued or waiting. I'm not sure why you don't believe this. The OP reported measurable delays. Therefore, it was unlikely the DASD was delivering optimal response times. ..snip I believe Response time does matter because it is a guide to the action required. It just should not be considered as the only metric that decides if IO is delaying an application. That's the key point putting us into substantial agreement. ..snip Ron -Original Message- From: IBM Mainframe Discussion List [mailto:ibm-m...@bama.ua.edu] On Behalf Of Hal Merritt Sent: Thursday, July 22, 2010 7:49 AM To: IBM-MAIN@bama.ua.edu Subject: Re: [IBM-MAIN] Any ROT for DASD Response time Short answer: yes. And you go on to agree. Let's take a look at the OP's issue: Not meeting SLA's. A logical first place to look is DASD, and we find 'delays'. What kind of delay and how long of a delay we don't know. The question is how long of a delay is normal and to be expected. A ROT for an acceptable range. I find it a little hard to believe that a DASD subsystem can deliver 0.3ms response times with measurable delays. Therefore, IMHO, any measurable delay should be investigated and, if feasible, reduced. To zero. Of course, I/O avoidance is almost always something we want to consider. But that does not address the OP's question. NOTICE: This electronic mail message and any files transmitted with it are intended exclusively for the individual or entity to which it is addressed. The message, together with any attachment, may contain confidential and/or privileged information. Any unauthorized review, use, printing, saving, copying, disclosure or distribution is strictly prohibited. If you have received this message in error, please immediately advise the sender by reply email and delete all copies. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: Any ROT for DASD Response time
See imbedded. Hal -Original Message- From: IBM Mainframe Discussion List [mailto:ibm-m...@bama.ua.edu] On Behalf Of Ron Hawkins Sent: Thursday, July 22, 2010 10:54 PM To: IBM-MAIN@bama.ua.edu Subject: Re: Any ROT for DASD Response time Hal, I should have been clear. The statement I disagreed with is: It does not really matter what the response times are, the issue is that they are too slow I'm not sure where I can agree with you, and I still fail to see how near perfect response time can be too slow. If the DASD is delivering near perfect response times, then it can't go faster. If that's still not fast enough, then one -must- address I/O reduction. War story: most loved online was hitting a ceiling and just wouldn't scale past a specific transaction load. The cause was traced to a single checkpoint dataset. The fastest hardware available was brought in but it was not fast enough. The very reluctant, politically powerful support team was forced to address the I/O to that dataset. A relatively minor change solved the problem. The point is that making application changes may meet with a -lot- of resistance, but ya gotta do whatcha gotta do. To be fair, -any- change to that application was a medium to high risk. Any volume averaging 0.3ms response time is pretty close to zero delay, where a delay is something queued or waiting. I'm not sure why you don't believe this. The OP reported measurable delays. Therefore, it was unlikely the DASD was delivering optimal response times. ..snip I believe Response time does matter because it is a guide to the action required. It just should not be considered as the only metric that decides if IO is delaying an application. That's the key point putting us into substantial agreement. ..snip Ron -Original Message- From: IBM Mainframe Discussion List [mailto:ibm-m...@bama.ua.edu] On Behalf Of Hal Merritt Sent: Thursday, July 22, 2010 7:49 AM To: IBM-MAIN@bama.ua.edu Subject: Re: [IBM-MAIN] Any ROT for DASD Response time Short answer: yes. And you go on to agree. Let's take a look at the OP's issue: Not meeting SLA's. A logical first place to look is DASD, and we find 'delays'. What kind of delay and how long of a delay we don't know. The question is how long of a delay is normal and to be expected. A ROT for an acceptable range. I find it a little hard to believe that a DASD subsystem can deliver 0.3ms response times with measurable delays. Therefore, IMHO, any measurable delay should be investigated and, if feasible, reduced. To zero. Of course, I/O avoidance is almost always something we want to consider. But that does not address the OP's question. NOTICE: This electronic mail message and any files transmitted with it are intended exclusively for the individual or entity to which it is addressed. The message, together with any attachment, may contain confidential and/or privileged information. Any unauthorized review, use, printing, saving, copying, disclosure or distribution is strictly prohibited. If you have received this message in error, please immediately advise the sender by reply email and delete all copies. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: Any ROT for DASD Response time
Short answer: yes. And you go on to agree. Let's take a look at the OP's issue: Not meeting SLA's. A logical first place to look is DASD, and we find 'delays'. What kind of delay and how long of a delay we don't know. The question is how long of a delay is normal and to be expected. A ROT for an acceptable range. I find it a little hard to believe that a DASD subsystem can deliver 0.3ms response times with measurable delays. Therefore, IMHO, any measurable delay should be investigated and, if feasible, reduced. To zero. Of course, I/O avoidance is almost always something we want to consider. But that does not address the OP's question. -Original Message- From: IBM Mainframe Discussion List [mailto:ibm-m...@bama.ua.edu] On Behalf Of Ron Hawkins Sent: Wednesday, July 21, 2010 11:29 PM To: IBM-MAIN@bama.ua.edu Subject: Re: Any ROT for DASD Response time Hal, Is that always true? If your application is getting 100% cache hits with 0.3ms response time, has a sub second service level, but issues 10,000 IO then there is no way you will meet your service level even though your response time is as near to perfect as it can be. The number of IO you can do can be as big a problem as the time the IO takes. Looking for some IO avoidance before handing over dollars for more cache, faster drives and SSD can will often result in improved Service Levels, while response stays the same, or even gets worse. It's all about reducing the net IO Time. For example a three level indexed KSDS can easily realize a 350-390% reduction in IO just by changing it to BLSR or SMB and using deferred writes. With that sort of saving would a DASD response time is 0.5ms or 10ms be important? Ron -Original Message- From: IBM Mainframe Discussion List [mailto:ibm-m...@bama.ua.edu] On Behalf Of Hal Merritt Sent: Wednesday, July 21, 2010 10:32 AM To: IBM-MAIN@bama.ua.edu Subject: Re: [IBM-MAIN] Any ROT for DASD Response time Point is that you are not meeting your SLA goals. If DASD response time is an issue, then it can be described as 'poor DASD performance'. It does not really matter what the response times are, the issue is that they are too slow. In this context a ROT for any delay would be zero, IMHO. HTH and good luck. -Original Message- From: IBM Mainframe Discussion List [mailto:ibm-m...@bama.ua.edu] On Behalf Of Lizette Koehler Sent: Wednesday, July 21, 2010 11:39 AM To: IBM-MAIN@bama.ua.edu Subject: Any ROT for DASD Response time We are running an EMC DMX4500 shared with the open system side. MVS has 9TB and the open side has 500TB. So they win. We have points in time where our production application (DB2 Stored Procedures with Native SQL) has slow responses. The DBAs are using RMF and TMON (DB2 and MVS) to isolate the problem. However, they see DFHSM and say thats the problem. I think I am seeing delays on the dasd the DB2 system uses in RMF. But I am not sure how to interpret the numbers. Is there a ROT that says if the delay rate is ? then the dasd is poor performaing? Thanks Lizette NOTICE: This electronic mail message and any files transmitted with it are intended exclusively for the individual or entity to which it is addressed. The message, together with any attachment, may contain confidential and/or privileged information. Any unauthorized review, use, printing, saving, copying, disclosure or distribution is strictly prohibited. If you have received this message in error, please immediately advise the sender by reply email and delete all copies. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html NOTICE: This electronic mail message and any files transmitted with it are intended exclusively for the individual or entity to which it is addressed. The message, together with any attachment, may contain confidential and/or privileged information. Any unauthorized review, use, printing, saving, copying, disclosure or distribution is strictly prohibited. If you have received this message in error, please immediately advise the sender by reply email and delete all copies. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: Any ROT for DASD Response time
Ask the DBA's why they say thats DFHSM causing the delay, are there recalls necessary for DB2 work ? What kind of delays do you see on the dasd volumes. What is causing the main delay according to RMF monitor 3 ? 2010/7/21 Lizette Koehler stars...@mindspring.com We are running an EMC DMX4500 shared with the open system side. MVS has 9TB and the open side has 500TB. So they win. We have points in time where our production application (DB2 Stored Procedures with Native SQL) has slow responses. The DBAs are using RMF and TMON (DB2 and MVS) to isolate the problem. However, they see DFHSM and say thats the problem. I think I am seeing delays on the dasd the DB2 system uses in RMF. But I am not sure how to interpret the numbers. Is there a ROT that says if the delay rate is ? then the dasd is poor performaing? Thanks Lizette -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html -- Met vriendelijke groeten/With kind regards Dick de Groot -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: Any ROT for DASD Response time
ROT is not important now, the main problem you are having a performance problem and not meeting your SLA's. Important is to identify what is causing that problem. 2010/7/21 Lizette Koehler stars...@mindspring.com We are running an EMC DMX4500 shared with the open system side. MVS has 9TB and the open side has 500TB. So they win. We have points in time where our production application (DB2 Stored Procedures with Native SQL) has slow responses. The DBAs are using RMF and TMON (DB2 and MVS) to isolate the problem. However, they see DFHSM and say thats the problem. I think I am seeing delays on the dasd the DB2 system uses in RMF. But I am not sure how to interpret the numbers. Is there a ROT that says if the delay rate is ? then the dasd is poor performaing? Thanks Lizette -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html -- Met vriendelijke groeten/With kind regards Dick de Groot -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: Any ROT for DASD Response time
Hal, I should have been clear. The statement I disagreed with is: It does not really matter what the response times are, the issue is that they are too slow I'm not sure where I can agree with you, and I still fail to see how near perfect response time can be too slow. Any volume averaging 0.3ms response time is pretty close to zero delay, where a delay is something queued or waiting. I'm not sure why you don't believe this. I've seen shorter response time with synthetic workloads designed to only use local memory in the channel interface, but that's hardly applicable to Lizette's problem. I believe Response time does matter because it is a guide to the action required. It just should not be considered as the only metric that decides if IO is delaying an application. In RMF III a cache hit is connect time only and counted as Using. It will not show up as a delay unless there is contention. Ron -Original Message- From: IBM Mainframe Discussion List [mailto:ibm-m...@bama.ua.edu] On Behalf Of Hal Merritt Sent: Thursday, July 22, 2010 7:49 AM To: IBM-MAIN@bama.ua.edu Subject: Re: [IBM-MAIN] Any ROT for DASD Response time Short answer: yes. And you go on to agree. Let's take a look at the OP's issue: Not meeting SLA's. A logical first place to look is DASD, and we find 'delays'. What kind of delay and how long of a delay we don't know. The question is how long of a delay is normal and to be expected. A ROT for an acceptable range. I find it a little hard to believe that a DASD subsystem can deliver 0.3ms response times with measurable delays. Therefore, IMHO, any measurable delay should be investigated and, if feasible, reduced. To zero. Of course, I/O avoidance is almost always something we want to consider. But that does not address the OP's question. -Original Message- From: IBM Mainframe Discussion List [mailto:ibm-m...@bama.ua.edu] On Behalf Of Ron Hawkins Sent: Wednesday, July 21, 2010 11:29 PM To: IBM-MAIN@bama.ua.edu Subject: Re: Any ROT for DASD Response time Hal, Is that always true? If your application is getting 100% cache hits with 0.3ms response time, has a sub second service level, but issues 10,000 IO then there is no way you will meet your service level even though your response time is as near to perfect as it can be. The number of IO you can do can be as big a problem as the time the IO takes. Looking for some IO avoidance before handing over dollars for more cache, faster drives and SSD can will often result in improved Service Levels, while response stays the same, or even gets worse. It's all about reducing the net IO Time. For example a three level indexed KSDS can easily realize a 350-390% reduction in IO just by changing it to BLSR or SMB and using deferred writes. With that sort of saving would a DASD response time is 0.5ms or 10ms be important? Ron -Original Message- From: IBM Mainframe Discussion List [mailto:ibm-m...@bama.ua.edu] On Behalf Of Hal Merritt Sent: Wednesday, July 21, 2010 10:32 AM To: IBM-MAIN@bama.ua.edu Subject: Re: [IBM-MAIN] Any ROT for DASD Response time Point is that you are not meeting your SLA goals. If DASD response time is an issue, then it can be described as 'poor DASD performance'. It does not really matter what the response times are, the issue is that they are too slow. In this context a ROT for any delay would be zero, IMHO. HTH and good luck. -Original Message- From: IBM Mainframe Discussion List [mailto:ibm-m...@bama.ua.edu] On Behalf Of Lizette Koehler Sent: Wednesday, July 21, 2010 11:39 AM To: IBM-MAIN@bama.ua.edu Subject: Any ROT for DASD Response time We are running an EMC DMX4500 shared with the open system side. MVS has 9TB and the open side has 500TB. So they win. We have points in time where our production application (DB2 Stored Procedures with Native SQL) has slow responses. The DBAs are using RMF and TMON (DB2 and MVS) to isolate the problem. However, they see DFHSM and say thats the problem. I think I am seeing delays on the dasd the DB2 system uses in RMF. But I am not sure how to interpret the numbers. Is there a ROT that says if the delay rate is ? then the dasd is poor performaing? Thanks Lizette NOTICE: This electronic mail message and any files transmitted with it are intended exclusively for the individual or entity to which it is addressed. The message, together with any attachment, may contain confidential and/or privileged information. Any unauthorized review, use, printing, saving, copying, disclosure or distribution is strictly prohibited. If you have received this message in error, please immediately advise the sender by reply email and delete all copies. -- For IBM-MAIN subscribe / signoff / archive
Any ROT for DASD Response time
We are running an EMC DMX4500 shared with the open system side. MVS has 9TB and the open side has 500TB. So they win. We have points in time where our production application (DB2 Stored Procedures with Native SQL) has slow responses. The DBAs are using RMF and TMON (DB2 and MVS) to isolate the problem. However, they see DFHSM and say thats the problem. I think I am seeing delays on the dasd the DB2 system uses in RMF. But I am not sure how to interpret the numbers. Is there a ROT that says if the delay rate is ? then the dasd is poor performaing? Thanks Lizette -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: Any ROT for DASD Response time
Point is that you are not meeting your SLA goals. If DASD response time is an issue, then it can be described as 'poor DASD performance'. It does not really matter what the response times are, the issue is that they are too slow. In this context a ROT for any delay would be zero, IMHO. HTH and good luck. -Original Message- From: IBM Mainframe Discussion List [mailto:ibm-m...@bama.ua.edu] On Behalf Of Lizette Koehler Sent: Wednesday, July 21, 2010 11:39 AM To: IBM-MAIN@bama.ua.edu Subject: Any ROT for DASD Response time We are running an EMC DMX4500 shared with the open system side. MVS has 9TB and the open side has 500TB. So they win. We have points in time where our production application (DB2 Stored Procedures with Native SQL) has slow responses. The DBAs are using RMF and TMON (DB2 and MVS) to isolate the problem. However, they see DFHSM and say thats the problem. I think I am seeing delays on the dasd the DB2 system uses in RMF. But I am not sure how to interpret the numbers. Is there a ROT that says if the delay rate is ? then the dasd is poor performaing? Thanks Lizette NOTICE: This electronic mail message and any files transmitted with it are intended exclusively for the individual or entity to which it is addressed. The message, together with any attachment, may contain confidential and/or privileged information. Any unauthorized review, use, printing, saving, copying, disclosure or distribution is strictly prohibited. If you have received this message in error, please immediately advise the sender by reply email and delete all copies. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: Any ROT for DASD Response time
Your DBA's are not even close. Unless the DB2 address spaces are waiting for DFHSM services(unlikely), faggedabouditt! Your normal tuning should insulate DB2 from any vagaries of DFHSM processing. IMO the most likely cause (if there is DASD interference) would be in the cache. I don't know if EMC has the ability to fence the cache for workload separation(open systems vs. z/OS). Check the cache hit ratio's for the time periods in question. There could also be some z/os image to z/os image type interference. There *might* be some backend interference on the DMX4500, but this type of analysis would most likely require the assistance of EMC to identify the problem (if any). ESCON connections w/cache should provide 3-5 ms response time. Not sure about FICON/FIBRE CHANNEL. The RMF spreadsheet reporter has some very good information if you have it available. The DASD code mirrors some work Tom Berevtas had done prior to his retirement from IBM. I have taken his course and was quite impressed with his approach and the tuning results based on that approach. Performance Associates( http://www.perfassoc.com/) might also have some good info. HTH, snip We are running an EMC DMX4500 shared with the open system side. MVS has 9TB and the open side has 500TB. So they win. We have points in time where our production application (DB2 Stored Procedures with Native SQL) has slow responses. The DBAs are using RMF and TMON (DB2 and MVS) to isolate the problem. However, they see DFHSM and say thats the problem. I think I am seeing delays on the dasd the DB2 system uses in RMF. But I am not sure how to interpret the numbers. Is there a ROT that says if the delay rate is ? then the dasd is poor performaing? /snip -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: Any ROT for DASD Response time
So they win. They don't have to ... I seem to remember some discussions here on IBM-Main about this subject not too long ago. And, IIRC, the outcome was to dedicate some of the cache memory in the DASD box to the mainframe, in order to preserve I/O performance. Ask EMC to review internal performance stats in that DASD box and see what they can do about setting aside enough cache memory for the mainframe. HTH Regards, Ulrich Krueger -Original Message- From: IBM Mainframe Discussion List [mailto:ibm-m...@bama.ua.edu] On Behalf Of Lizette Koehler Sent: Wednesday, July 21, 2010 09:39 To: IBM-MAIN@bama.ua.edu Subject: Any ROT for DASD Response time We are running an EMC DMX4500 shared with the open system side. MVS has 9TB and the open side has 500TB. So they win. We have points in time where our production application (DB2 Stored Procedures with Native SQL) has slow responses. The DBAs are using RMF and TMON (DB2 and MVS) to isolate the problem. However, they see DFHSM and say thats the problem. I think I am seeing delays on the dasd the DB2 system uses in RMF. But I am not sure how to interpret the numbers. Is there a ROT that says if the delay rate is ? then the dasd is poor performaing? Thanks Lizette -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: Any ROT for DASD Response time
On Wed, 21 Jul 2010 12:36:38 -0500, Staller, Allan allan.stal...@kbm1.com wrote: I don't know if EMC has the ability to fence the cache for workload separation (open systems vs. z/OS). That would be a no. We now live happily together here again on a VMAX. Once upon a time in Symmetrix land we tried having both on the same box, but the technology just didn't live up to it's billing. We went to seperate boxes until March of this year, where management saw it fit to drop license and maintenance down to 1 box[1]*. Just after conversion of the mainframe to the VMAX, the existing open systems started complaining of higher I/O times. Our cache hits were in the 80 percentile range IIRC. Once where open systems data had the entire range of cache to feed, a lot would not drop out. Then the big bad mainframe, the black box of rebound, came in and boxed out open systems data that was not referenced quite as often. *[1] Actually, there is two boxes, one for production and one for DR. So essentially we went from 4 licenses down to two. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: Any ROT for DASD Response time
YMMV! snip That would be a no. We now live happily together here again on a VMAX. Once upon a time in Symmetrix land we tried having both on the same box, but the technology just didn't live up to it's billing. We went to seperate boxes until March of this year, where management saw it fit to drop license and maintenance down to 1 box[1]*. Just after conversion of the mainframe to the VMAX, the existing open systems started complaining of higher I/O times. Our cache hits were in the 80 percentile range IIRC. Once where open systems data had the entire range of cache to feed, a lot would not drop out. Then the big bad mainframe, the black box of rebound, came in and boxed out open systems data that was not referenced quite as often. /snip -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: Any ROT for DASD Response time
In this context a ROT for any delay would be zero, IMHO. That, of course, is totally unrealistic. 2-5ms/IO is reasonable, IMO. - I'm a SuperHero with neither powers, nor motivation! Kimota! -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: Any ROT for DASD Response time
In a message dated 7/21/2010 3:02:23 P.M. Central Daylight Time, eamacn...@yahoo.ca writes: That, of course, is totally unrealistic. 2-5ms/IO is reasonable, IMO. The folks over at _www.perfassoc.com_ (http://www.perfassoc.com) do this for a living. They got White Papers on different vendors and models. For $$$ they'll help you more. SHARE and Redbooks are good trolling grounds. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: Any ROT for DASD Response time
The folks over at _www.perfassoc.com_ (http://www.perfassoc.com) do this for a living. Yes, I know that. But, I was responding to the poster who said delay had to be ZERO. IMPOSSIBLE! - I'm a SuperHero with neither powers, nor motivation! Kimota! -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: Any ROT for DASD Response time
Hal, Is that always true? If your application is getting 100% cache hits with 0.3ms response time, has a sub second service level, but issues 10,000 IO then there is no way you will meet your service level even though your response time is as near to perfect as it can be. The number of IO you can do can be as big a problem as the time the IO takes. Looking for some IO avoidance before handing over dollars for more cache, faster drives and SSD can will often result in improved Service Levels, while response stays the same, or even gets worse. It's all about reducing the net IO Time. For example a three level indexed KSDS can easily realize a 350-390% reduction in IO just by changing it to BLSR or SMB and using deferred writes. With that sort of saving would a DASD response time is 0.5ms or 10ms be important? Ron -Original Message- From: IBM Mainframe Discussion List [mailto:ibm-m...@bama.ua.edu] On Behalf Of Hal Merritt Sent: Wednesday, July 21, 2010 10:32 AM To: IBM-MAIN@bama.ua.edu Subject: Re: [IBM-MAIN] Any ROT for DASD Response time Point is that you are not meeting your SLA goals. If DASD response time is an issue, then it can be described as 'poor DASD performance'. It does not really matter what the response times are, the issue is that they are too slow. In this context a ROT for any delay would be zero, IMHO. HTH and good luck. -Original Message- From: IBM Mainframe Discussion List [mailto:ibm-m...@bama.ua.edu] On Behalf Of Lizette Koehler Sent: Wednesday, July 21, 2010 11:39 AM To: IBM-MAIN@bama.ua.edu Subject: Any ROT for DASD Response time We are running an EMC DMX4500 shared with the open system side. MVS has 9TB and the open side has 500TB. So they win. We have points in time where our production application (DB2 Stored Procedures with Native SQL) has slow responses. The DBAs are using RMF and TMON (DB2 and MVS) to isolate the problem. However, they see DFHSM and say thats the problem. I think I am seeing delays on the dasd the DB2 system uses in RMF. But I am not sure how to interpret the numbers. Is there a ROT that says if the delay rate is ? then the dasd is poor performaing? Thanks Lizette NOTICE: This electronic mail message and any files transmitted with it are intended exclusively for the individual or entity to which it is addressed. The message, together with any attachment, may contain confidential and/or privileged information. Any unauthorized review, use, printing, saving, copying, disclosure or distribution is strictly prohibited. If you have received this message in error, please immediately advise the sender by reply email and delete all copies. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: Any ROT for DASD Response time
All, I recommend Lizette have a look at the papers do you have one second response time and with Everything You Know is Wrong, and decide from empirical observation what a good response time is for the workload in question. For example, if this is predominantly read CA-IDMS or IMS DEDB, workloads that have notoriously poor locality of reference because of key hashing, then one would expect response time to be in the 5-15ms range, depending on how busy spindles are and how the workload is short or long stroked. If this CICS/VSAM application with a heavy random write load, than with a 4KB CISZ it would perfectly reasonable to expect response time to be less than 0.5ms with near zero disconnect time if the workload has been laid out to avoid destage skew. However turn on synchronous remote copy over 200km and that 0.5ms will become more like 2ms. And we could chat for hours about the affect of busy Initiator and Target microprocessors on those long, chained IO that DFSORT and SYNCSORT like to do... It's IO elongation at its best (or worse as the case may be). So while one can come up with a rule of thumb for DASD response time, it does require some categorization of the IO generated by the workload, and an understanding of the underlying Disk topology that will process the IO before one settles on the right rule of thumb. It will probably be somewhere between what you want and what you are getting. Ron -Original Message- From: IBM Mainframe Discussion List [mailto:ibm-m...@bama.ua.edu] On Behalf Of Ed Finnell Sent: Wednesday, July 21, 2010 8:29 PM To: IBM-MAIN@bama.ua.edu Subject: Re: [IBM-MAIN] Any ROT for DASD Response time In a message dated 7/21/2010 3:02:23 P.M. Central Daylight Time, eamacn...@yahoo.ca writes: That, of course, is totally unrealistic. 2-5ms/IO is reasonable, IMO. The folks over at _www.perfassoc.com_ (http://www.perfassoc.com) do this for a living. They got White Papers on different vendors and models. For $$$ they'll help you more. SHARE and Redbooks are good trolling grounds. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
