Re: Short term paging spike
We see such spikes caused by the WMQI Broker (or what is called these days). At startup, it allocates a huge amount of storage (0.6 GB) per address space (4 or 5), which is paged out for 90% after some time. Caused us an aux storage shortage. And a subsequent haggle with IBM, as their LE defaults are a mess (HEAP, HEAPPOOLS). Ever noticed the amount of FREF pages? Regards, Barbara Nitz -- 5 GB Mailbox, 50 FreeSMS http://www.gmx.net/de/go/promail +++ GMX - die erste Adresse für Mail, Message, More +++ -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: Short term paging spike
Ted MacNEIL [EMAIL PROTECTED] wrote in message news:[EMAIL PROTECTED]... This was a while ago in a different age, prior to WLM and big real storage, but I suspect the same situation could be a possibility. .. This can still happen in GOAL MODE. At the same time IBM introduced CPU-Critical, they introduced MEMORY-Critical. This was due to a large bank in the US having exactly this issue. The key is to identify the culprit/victim and make them MEMORY-Critical. Only if this is the problem! The original question was about paging spikes, which by itsself are usually not a problem for the system. If the spikes are for an application, that suffers from it, it could be helped by making it Memory-critical. Kees. ** For information, services and offers, please visit our web site: http://www.klm.com. This e-mail and any attachment may contain confidential and privileged material intended for the addressee only. If you are not the addressee, you are notified that no part of the e-mail or any attachment may be disclosed, copied or distributed, and that any other action related to this e-mail or attachment is strictly prohibited, and may be unlawful. If you have received this e-mail by error, please notify the sender immediately by return e-mail, and delete this message. Koninklijke Luchtvaart Maatschappij NV (KLM), its subsidiaries and/or its employees shall not be liable for the incorrect or incomplete transmission of this e-mail or any attachments, nor responsible for any delay in receipt. ** -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: Short term paging spike
Bob, Closing/Opening a bunch of files on CICS Region that has been relatively dormant all night will bring this on. Some one else mentioned the Monday Morning affect, and this is part of it - but every morning. The Page-outs probably go unnoticed because they are spread over a longer period of time, or after the first page-out they are unchanged pages and move through a bit flick. Ron -Original Message- From: IBM Mainframe Discussion List [mailto:[EMAIL PROTECTED] On Behalf Of Bob Sent: Wednesday, 21 September 2005 4:59 AM To: IBM-MAIN@BAMA.UA.EDU Subject: Short term paging spike I am seeing short term paging spikes (zOS 1.4 system) at around 6am. No dumps happening, just a rapid drop in AFQ and then a spike of 500 pages / sec... for a about 1 min. Can anyone recommend how to find out who the causer is? Do SMF records capture this? The stuff I see doesn't have the granularity to find the culprit(s). -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: Short term paging spike
To all, Thanks to all for good suggestions. I'e been looking at this some more and the fact is that we were getting home grown page rate alerts due to a couple factors. 1) Automation kicks off a bunch of stuff at 0500, 0530, 0600. 2) Our enterprising automation guy added a pagerate probe that kicks off at the same times. It's not dynamic, just a snapshot he takes every 1/2 hour. False positives. Mark, I took a look at RMF III and that report shows current aux store usage by asid, but only goes back an hour or so. I am still wondering the best way for me to shoot a ... say 5 min slowdown caused by a few ASIDS running paging off the charts, a few days after the fact. Does Candle Omegamon epilog keep that kind of detail? -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: Short term paging spike
In a message dated 9/22/2005 7:54:59 A.M. Central Standard Time, [EMAIL PROTECTED] writes: say 5 min slowdown caused by a few ASIDS running paging off the charts, a few days after the fact. Does Candle Omegamon epilog keep that kind of detail? RMF will report at RMF interval. Candle has numerous knobs to set. If you set them too close or too frequently you'll see more CPU eaten up in the monitoring than in the programs. Most of them are set in profiles for each classes of service. It may be that spreading the spike over 15 minute interval would decrease the big spikes. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: Short term paging spike
This is a PRIVATE message. If you are not the intended recipient, please delete without copying and kindly advise us by e-mail of the mistake in delivery. NOTE: Regardless of content, this e-mail shall not operate to bind CSC to any order or other contract unless pursuant to explicit written agreement or government initiative expressly permitting the use of e-mail for such purpose. Bob H rhensel @TIAA-CREF.ORG To: IBM-MAIN@BAMA.UA.EDU Sent by: IBM cc: MainframeSubject: Re: Short term paging spike Discussion List IBM-MAIN 09/22/2005 08:54 AM Please respond to IBM Mainframe Discussion List snip I took a look at RMF III and that report shows current aux store usage by asid, but only goes back an hour or so. /unsnip I get to watch over a number of LPARs, each running RMF III configured to record to 4-6 150-cylinder datasets. On every LPAR, RMF III lookback timespan is at least 2 days. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: Short term paging spike
I haven't worked with EPILOG for a while but I don't believe that you can get that kind of detail unless it has changed. You could hiper-link to generic aux. storage information from specific screen sets but I don't remember seeing the detail you may need to shoot this. I currently use Mainview, with a significant back end history database, and it has an RMF III look alike menu which shows what tasks do paging over set time intervals. I'm sure that RMF III can present this data but I'm not familiar enough with RMF III to give you the exact information. There's got to be a way to set up a new RMF III database and repoint it, at the desired time, to get the data. Or increase the RMF III database to get more than 1 hour of data. Bob H [EMAIL PROTECTED] Sent by: IBM Mainframe Discussion List IBM-MAIN@BAMA.UA.EDU 09/22/2005 08:54 AM Please respond to IBM Mainframe Discussion List To: IBM-MAIN@BAMA.UA.EDU cc: Subject:Re: Short term paging spike To all, Thanks to all for good suggestions. I'e been looking at this some more and the fact is that we were getting home grown page rate alerts due to a couple factors. 1) Automation kicks off a bunch of stuff at 0500, 0530, 0600. 2) Our enterprising automation guy added a pagerate probe that kicks off at the same times. It's not dynamic, just a snapshot he takes every 1/2 hour. False positives. Mark, I took a look at RMF III and that report shows current aux store usage by asid, but only goes back an hour or so. I am still wondering the best way for me to shoot a ... say 5 min slowdown caused by a few ASIDS running paging off the charts, a few days after the fact. Does Candle Omegamon epilog keep that kind of detail? -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: Short term paging spike
99 @ LPAR @ 1500 TRKS = one week Data available from 09/16/05 05.47.00 to 09/22/05 10.14.00 (present) which is typical of the 7 to 8 days we usually see depends a little on activity and has shrunk some with the recent addition of several very large storage processors which also chewed up a good deal of the available addressing range. RMF III data is a god send. I happily trade 943665 Tracks of DASD space for the 60 second detail on all these partitions. It saves me in many cases from needing to run batch reports or dump data and I am able to provide answers while on the phone with someone. Best Regards, Sam Knutson, GEICO Performance and Availability Management mailto:[EMAIL PROTECTED] (office) 301.986.3574 Five card stud, nothing wild. And the sky's the limit. - Picard, uttering the last line of the TV series, Star Trek: The Next Generation, (All Good Things...) -Original Message- snip I took a look at RMF III and that report shows current aux store usage by asid, but only goes back an hour or so. /unsnip I get to watch over a number of LPARs, each running RMF III configured to record to 4-6 150-cylinder datasets. On every LPAR, RMF III lookback timespan is at least 2 days. This email/fax message is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution of this email/fax is prohibited. If you are not the intended recipient, please destroy all paper and electronic copies of the original message. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: Short term paging spike
On Thu, 22 Sep 2005 07:54:51 -0500, Bob H [EMAIL PROTECTED] wrote: Mark, I took a look at RMF III and that report shows current aux store usage by asid, but only goes back an hour or so. I am still wondering the best way for me to shoot a ... say 5 min slowdown caused by a few ASIDS running paging off the charts, a few days after the fact. Does Candle Omegamon epilog keep that kind of detail? It sounds like you are using the default RMF III member (ERBRMF04) from SYS1.IBM.PARMLIB. You need to define history data sets and add them to the RMF III parm member. There is a rexx exec in in SYS1.SERBCLS to help you do this: ERBVSDEF VSAM_DSN VSAMVOL(VOLUME) TRACKS(NUM_TRACKS) I have 18 data sets that are 50 CYLs each. Depending on which LPAR they hold anywhere from 40 to 72 hours worth of data. This depends on the MINTIME setting also in ERBRMFxx. See the RMF manuals for more details. If you are curious about why so many small data sets, check the archives and the section on defining the data sets in the RMF user guide. Mark -- Mark Zelden Sr. Software and Systems Architect - z/OS Team Lead Zurich North America and Farmers Insurance Group mailto: [EMAIL PROTECTED] Systems Programming expert at http://Search390.com/ateExperts/ Mark's MVS Utilities: http://home.flash.net/~mzelden/mvsutil.html -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: Short term paging spike
You can use RMF II to monitor the system and write to SMF. You need ASD(A,A). Run the post processor and look for non-zero PIN RT. Set the interval for around one minute. Depending on the size of your shop, this should add 1-3% to your SMF data. I've done this in the past. -teD In God we Trust! All others bring data! -- W. Edwards Deming -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: Short term paging spike
I was thinking about a similar cause. We see such spikes caused by the WMQI Broker (or what is called these days). At startup, it allocates a huge amount of storage (0.6 GB) per address space (4 or 5), which is paged out for 90% after some time. A Stop/Start of the application frees the paged out storage, so does not provide much relief for Central Storage during Stop, but subsequent Start allocates again 4 or 5 times 0.6 GB. The system needs some time (and paging) to establish a new stable situtation. Kees. Shane Ginnane [EMAIL PROTECTED] wrote in message news:[EMAIL PROTECTED]... Last time I noticed something like this, we put it down to early starters. Online regions not getting used overnight, wound up getting (heavily) trimmed. First arrival in the morning whacks the Enter key and (probably) strolls off to get a coffee. Different office, similar scenario, but hitting a different region. Mad scramble for swap-in of large working sets. Once they are in and active, no paging. This was a while ago in a different age, prior to WLM and big real storage, but I suspect the same situation could be a possibility. Shane ... I am seeing short term paging spikes (zOS 1.4 system) at around 6am. No dumps happening, just a rapid drop in AFQ and then a spike of 500 pages / sec... for a about 1 min. Can anyone recommend how to find out who the causer is? Do SMF records capture this? The stuff I see doesn't have the granularity to find the culprit(s). ** For information, services and offers, please visit our web site: http://www.klm.com. This e-mail and any attachment may contain confidential and privileged material intended for the addressee only. If you are not the addressee, you are notified that no part of the e-mail or any attachment may be disclosed, copied or distributed, and that any other action related to this e-mail or attachment is strictly prohibited, and may be unlawful. If you have received this e-mail by error, please notify the sender immediately by return e-mail, and delete this message. Koninklijke Luchtvaart Maatschappij NV (KLM), its subsidiaries and/or its employees shall not be liable for the incorrect or incomplete transmission of this e-mail or any attachments, nor responsible for any delay in receipt. ** -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: Short term paging spike
SMF 30 could also be used (working set, but it would probably not be my first place to look. ... Maybe not; it does have PIN data, though -- efficient if you are using interval (of your choice) data. -teD In God we Trust! All others bring data! -- W. Edwards Deming -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: Short term paging spike
This was a while ago in a different age, prior to WLM and big real storage, but I suspect the same situation could be a possibility. ... This can still happen in GOAL MODE. At the same time IBM introduced CPU-Critical, they introduced MEMORY-Critical. This was due to a large bank in the US having exactly this issue. The key is to identify the culprit/victim and make them MEMORY-Critical. -teD In God we Trust! All others bring data! -- W. Edwards Deming -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: Short term paging spike
On Tue, 20 Sep 2005 15:58:37 -0500, Bob [EMAIL PROTECTED] wrote: I am seeing short term paging spikes (zOS 1.4 system) at around 6am. No dumps happening, just a rapid drop in AFQ and then a spike of 500 pages / sec... for a about 1 min. Can anyone recommend how to find out who the causer is? Do SMF records capture this? The stuff I see doesn't have the granularity to find the culprit(s). Do you run RMF III? The STORF command might show you the culprit, and you don't have to be there to watch it real time. Otherwise, monitors like Omegamon, TMON, MXI, or even a SDSF DA display (with SORT REAL D might show you. SMF 30 could also be used (working set, but it would probably not be my first place to look. Mark -- Mark Zelden Sr. Software and Systems Architect - z/OS Team Lead Zurich North America and Farmers Insurance Group mailto: [EMAIL PROTECTED] Systems Programming expert at http://Search390.com/ateExperts/ Mark's MVS Utilities: http://home.flash.net/~mzelden/mvsutil.html -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: Short term paging spike
Last time I noticed something like this, we put it down to early starters. Online regions not getting used overnight, wound up getting (heavily) trimmed. First arrival in the morning whacks the Enter key and (probably) strolls off to get a coffee. Different office, similar scenario, but hitting a different region. Mad scramble for swap-in of large working sets. Once they are in and active, no paging. This was a while ago in a different age, prior to WLM and big real storage, but I suspect the same situation could be a possibility. Shane ... I am seeing short term paging spikes (zOS 1.4 system) at around 6am. No dumps happening, just a rapid drop in AFQ and then a spike of 500 pages / sec... for a about 1 min. Can anyone recommend how to find out who the causer is? Do SMF records capture this? The stuff I see doesn't have the granularity to find the culprit(s). -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
