help
NTAC:3NS-20 -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Recommendations for number of pagesets in MQ
NTAC:3NS-20 We are currently running MQ V7.1 (z/OS 2.2) and plan to migrate to V9 early next year. A couple of weeks ago we had an issue where a application 'audit queue' residing in pageset 01, along with the dynamic command queues, filled up the pageset due to an application error. This caused us to not be able to use the MQ ISPF interface or run batch jobs containing MQ commands. We could still issue commands via SDSF, and our response was to empty out and move this audit queue to pageset 02. We then found over 1400 orphaned dynamic command queues, 120+ with commands queued in them. We cleared the command queues manually and deleted them and the issue was resolved. IBM put some recommendations into the PMR around pageset guidelines, and we are looking to redo our pagesets to something that protects us against issues like this. We currently have 6 pagesets. Issue #1 is to move local queues out of pageset 01 so that the dynamic command queues don't get impacted by an application error again. We will be doing this. We have created two new pagesets (06 and 07) with the intention of isolating the audit queue into one and moving another audit queue that was discovered into the other, however, the application team made a change to stop using the audit queue so for now this is not an emergency. IBM recommended that we change the program that writes to these audit queues to use a database instead of MQ for them. I've passed that recommendation on. Does anyone have any suggestions for the layout of MQ pagesets? How many is too many? Why is it too many? Do other folks isolate application queues into their own pagesets, or are they intermingled by type, function or some other way? Thanks in advance! Jim -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: Recommendations for RECOVERY options
NTAC:3NS-20 IBM told me about the default. It is only changeable when you use SCOPE=CU. Didn't mean to upset anyone. Also didn't account for it being a non-work week for many. -Original Message- From: IBM Mainframe Discussion List [mailto:IBM-MAIN@LISTSERV.UA.EDU] On Behalf Of Tom Marchant Sent: Thursday, December 29, 2016 12:50 PM To: IBM-MAIN@LISTSERV.UA.EDU Subject: Re: Recommendations for RECOVERY options On Thu, 29 Dec 2016 13:45:53 +, James Peddycord wrote: >In this forum with so many people who have so many opinions, I can't >believe that nobody is offering suggestions for the RECOVERY parameter >when asked. This less than one day after asking the initial question. If you want to piss people off and get ignored, a good way to do it is to assume that you are entitled to a rapid response to your question. Or even any response at all. The people on this list are here voluntarily. We are here to learn and to offer help where we can. AFAIK, even the people from IBM who are here do so voluntarily. There are no Service Level Agreements, and no support contracts requiring a particular response time to your questions. This particular question is a rather esoteric one, and perhaps not many of us have experience in this particular area. I, for one, do not. You didn't even say that you were looking for the IOS Recovery options. I suppose we should have been smart enough to figure that out. >The default is: >RECOVERY,PATH_SCOPE=DEVICE,PATH_INTERVAL=10,PATH_THRESHOLD=100 Is it? According to the manual, PATH_INTERVAL and PATH_THRESHOLD can only be specified with SCOPE=CU. D IOS,RECOVERY will only display PATH_INTERVAL and PATH_THRESHOLD when SCOPE=CU. https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ibm.com_support _knowledgecenter_en_SSLTBW-5F2.2.0_com.ibm.zos.v2r2.ieae200_ieae200362.h tm=DgIFaQ=K5gMqH44tVpW9Mb7NvpzqAFAhrpSdUITR819D8huNsU=JBknay_mAJnJ KiefH4EC1w=53xrR1GIbE_QgUP29ajWREBmhze6wlPN2cNw-sXMfAs=B2f6alOR-dsWS Q3mqmW8ax4nimxv_xD4-OGCYW2Wr-4= https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ibm.com_support _knowledgecenter_en_SSLTBW-5F2.2.0_com.ibm.zos.v2r2.ieae200_ieae200360.h tm=DgIFaQ=K5gMqH44tVpW9Mb7NvpzqAFAhrpSdUITR819D8huNsU=JBknay_mAJnJ KiefH4EC1w=53xrR1GIbE_QgUP29ajWREBmhze6wlPN2cNw-sXMfAs=e-iEnhLMp2ggY jboHmbKq657V4374ig-SknztmfN3r8= -- Tom Marchant -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: Recommendations for RECOVERY options
NTAC:3NS-20 Thanks for the suggestion. I will repeat the question in a couple of weeks. Jim -Original Message- From: IBM Mainframe Discussion List [mailto:IBM-MAIN@LISTSERV.UA.EDU] On Behalf Of Elardus Engelbrecht Sent: Thursday, December 29, 2016 8:14 AM To: IBM-MAIN@LISTSERV.UA.EDU Subject: Re: Recommendations for RECOVERY options James Peddycord wrote: >In this forum with so many people who have so many opinions, I can't believe that nobody is offering suggestions for the RECOVERY parameter when asked. Perhaps these persons with that specific hardware combination are still on holiday. Or if someone has this specific combination like yours indeed reads your message, but never experieneced this specific bad cable problem and perhaps can't contribute. Others like me are working on other things like RACF (myself), general z/OS + JES2 + TSO/ISPF matters (also myself). There are programmers in various languages also active here. Now and then you get some one with problems with a specific product like DFDSS, DFSORT, CICS, DB2 etc. Please note: I was previously responsible for storage (SMS/HSM) and managing hardware (3380 and later 3390, 3490 tapes and Magstar, etc.). I have some experience with ESCON, but not with Ficon, but I do read ALL posts on IBM, except those from a madman earlier this month. I humbly suggest that you repeat your question in second week of January 2017 when most are then hopefully back from their holidays. Good luck in finding a good solution. Groete / Greetings Elardus Engelbrecht -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: Recommendations for RECOVERY options
NTAC:3NS-20 Our SAN people said they saw no errors on the switch (do I believe this? IDK) , which is dedicated to the mainframe. The problem was a bad cable between the switch and the storage, which both mainframes share, so there was an issue on one CHPID on each system. In this forum with so many people who have so many opinions, I can't believe that nobody is offering suggestions for the RECOVERY parameter when asked. The default is: RECOVERY,PATH_SCOPE=DEVICE,PATH_INTERVAL=10,PATH_THRESHOLD=100 This is what caused our pain. z/OS has to see 100 errors per minute for 10 minutes before taking the path off of a single device. This was happening to every device. On our test system I started with: RECOVERY,PATH_SCOPE=CU,PATH_INTERVAL=5,PATH_THRESHOLD=50 z/OS has to see 50 errors per minute for 5 minutes before taking the path off of every device in the LCU. I was hoping to see some real world examples that work for others. Jim -Original Message- From: IBM Mainframe Discussion List [mailto:IBM-MAIN@LISTSERV.UA.EDU] On Behalf Of Alan (GMAIL) Watthey Sent: Wednesday, December 28, 2016 11:55 PM To: IBM-MAIN@LISTSERV.UA.EDU Subject: Re: Recommendations for RECOVERY options Jim, As the network guy who looks after the SAN I would not be expecting my z/OS guys to do anything in this situation. In fact z/OS cannot see our whole SAN as I have other things on it (eg. ISLs, backend tapes). Fortunately, the Brocade switches are dedicated to the mainframes and devices used by the mainframes here. The Brocade will see issues that it recovers from well before z/OS sees anything. Of course, I don't know exactly what your problem was and what your Brocade would have seen but I'd suggest the bottleneckmon command to detect latency issues. Running the porterrshow command from time to time would also give you a good idea as to whether your SFPs and fibres are performing as well as they should. If you have the Fabric Vision/Watch licenses then you do fancy stuff like fence errant ports before they impact anything. Regards, Alan Watthey -Original Message- From: James Peddycord [mailto:j...@ntrs.com] Sent: 28 December 2016 4:56 pm Subject: Recommendations for RECOVERY options NTAC:3NS-20 We had a situation with a bad cable that resulted in a huge performance impact due to the default way that z/OS (we are at 1.13) handles error recovery on Ficon paths. The symptoms were many (thousands) of IOS050I messages in the task's joblog, followed by an IOS450E message, which took the path offline to a single device. This was happening for every device (around 3000) that the affected path was attached to. As soon as I saw the messages I configured the CHPID offline and the problem stopped. We have put in automation that will immediately configure a CHPID offline as soon as a single IOS450E message is detected, and now I am experimenting with RECOVERY options. IBM recommended to set RECOVERY,PATH_SCOPE=CU, set the PATH_INTERVAL to 1 and leave PATH_THRESHOLD=10, and adjust from there. Due to the paperwork involved with making any change in our environment, I would like to implement this with a minimum of 'adjustment'. Does anyone have any recommendations? We are running on z13s, 16G Ficon through Brokade switches to IBM DS88xx DASD. Thanks, Jim -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: Recommendations for RECOVERY options
NTAC:3NS-20 The performance impact was slow I/O to that group of devices, one at a time, until the path was taken offline. To the users, transactions that usually take less than a second were taking in some cases minutes. Bad enough to be considered an 'outage' from the user's perspective. No issue with console flooding. We have been suppressing IOS050I from the console for a long time. Jim -Original Message- From: IBM Mainframe Discussion List [mailto:IBM-MAIN@LISTSERV.UA.EDU] On Behalf Of Jousma, David Sent: Wednesday, December 28, 2016 8:03 AM To: IBM-MAIN@LISTSERV.UA.EDU Subject: Re: Recommendations for RECOVERY options What specifically was the performance impact? The loss of the ficon channel and reduced i/o bandwidth? Or was it the console message flooding? If the latter, implementing Message Flood automation will stop the flooding of messages. It is pretty easy to implement. Dave _ Dave Jousma Manager Mainframe Engineering, Assistant Vice President david.jou...@53.com 1830 East Paris, Grand Rapids, MI 49546 MD RSCB2H p 616.653.8429 f 616.653.2717 -Original Message- From: IBM Mainframe Discussion List [mailto:IBM-MAIN@LISTSERV.UA.EDU] On Behalf Of James Peddycord Sent: Wednesday, December 28, 2016 8:56 AM To: IBM-MAIN@LISTSERV.UA.EDU Subject: Recommendations for RECOVERY options NTAC:3NS-20 We had a situation with a bad cable that resulted in a huge performance impact due to the default way that z/OS (we are at 1.13) handles error recovery on Ficon paths. The symptoms were many (thousands) of IOS050I messages in the task's joblog, followed by an IOS450E message, which took the path offline to a single device. This was happening for every device (around 3000) that the affected path was attached to. As soon as I saw the messages I configured the CHPID offline and the problem stopped. We have put in automation that will immediately configure a CHPID offline as soon as a single IOS450E message is detected, and now I am experimenting with RECOVERY options. IBM recommended to set RECOVERY,PATH_SCOPE=CU, set the PATH_INTERVAL to 1 and leave PATH_THRESHOLD=10, and adjust from there. Due to the paperwork involved with making any change in our environment, I would like to implement this with a minimum of 'adjustment'. Does anyone have any recommendations? We are running on z13s, 16G Ficon through Brokade switches to IBM DS88xx DASD. Thanks, Jim -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN This e-mail transmission contains information that is confidential and may be privileged. It is intended only for the addressee(s) named above. If you receive this e-mail in error, please do not read, copy or disseminate it in any manner. If you are not the intended recipient, any disclosure, copying, distribution or use of the contents of this information is prohibited. Please reply to the message immediately by informing the sender that the message was misdirected. After replying, please erase it from your computer system. Your assistance in correcting this error is appreciated. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Recommendations for RECOVERY options
NTAC:3NS-20 We had a situation with a bad cable that resulted in a huge performance impact due to the default way that z/OS (we are at 1.13) handles error recovery on Ficon paths. The symptoms were many (thousands) of IOS050I messages in the task's joblog, followed by an IOS450E message, which took the path offline to a single device. This was happening for every device (around 3000) that the affected path was attached to. As soon as I saw the messages I configured the CHPID offline and the problem stopped. We have put in automation that will immediately configure a CHPID offline as soon as a single IOS450E message is detected, and now I am experimenting with RECOVERY options. IBM recommended to set RECOVERY,PATH_SCOPE=CU, set the PATH_INTERVAL to 1 and leave PATH_THRESHOLD=10, and adjust from there. Due to the paperwork involved with making any change in our environment, I would like to implement this with a minimum of 'adjustment'. Does anyone have any recommendations? We are running on z13s, 16G Ficon through Brokade switches to IBM DS88xx DASD. Thanks, Jim -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: Mainframe systems programmer ID 'vaulting'
NTAC:3NS-20 Our people are also working with the vendor (same vendor) to get Reflections working so that logon's can occur automatically. For now have a workaround and can log in using Reflections manually. What will really complicate matters is that all of the systems programmers have multiple logonids to log on to multiple systems simultaneously. Management is not going to like the additional troubleshooting time that will be incurred if we have to go check out an ID for each system we want to log on to. -Original Message- From: IBM Mainframe Discussion List [mailto:IBM-MAIN@LISTSERV.UA.EDU] On Behalf Of Steve Sent: Wednesday, November 23, 2016 9:09 AM To: IBM-MAIN@LISTSERV.UA.EDU Subject: Re: Mainframe systems programmer ID 'vaulting' Once we have CyberArk being a useful tool, all daily work will be done with TN3270 under CyberArk. That will not happen for a while but it is coming. Welcome to how the FED is going to work and you MUST have a CAC/PIV to even get through the firewalls Steve Beaver This electronic mail (including any attachments) may contain information that is privileged, confidential, and/or otherwise protected from disclosure to anyone other than its intended recipient(s). Any dissemination or use of this electronic email or its contents (including any attachments) by persons other than the intended recipient(s) is strictly prohibited. If you have received this message in error, please notify us immediately by reply email so that we may correct our internal records. Please then delete the original message (including any attachments) in its entirety. Thank you -Original Message- From: "James Peddycord" <j...@ntrs.com> Sent: Wednesday, November 23, 2016 10:03am To: IBM-MAIN@LISTSERV.UA.EDU Subject: Re: Mainframe systems programmer ID 'vaulting' NTAC:3NS-20 We have a FIRECALL process now, but it's automated. The vaulting process has taken the place of this FIRECALL process for most applications teams. The systems programmers don't need to get a FIRECALL ID to update SYS1 datasets. We only need them when working with datasets that we don't ordinarily have access to, such as the datasets that come in a service pack that have HLQs that we don't have dataset rules for. We are currently vaulting a pool of logonids with god-like power for systems programmers to check out. When this is complete we will retire our old FIRECALL process. It's the next step, where our daily use TSO IDs may be vaulted, that scares us. Jim -Original Message- From: IBM Mainframe Discussion List [mailto:IBM-MAIN@LISTSERV.UA.EDU] On Behalf Of Lester, Bob Sent: Tuesday, November 22, 2016 4:04 PM To: IBM-MAIN@LISTSERV.UA.EDU Subject: Re: Mainframe systems programmer ID 'vaulting' For a while, about 15 years ago, we had "firecall" IDs. When you logged in, it prompted you for information that, in turn, updated RACF with Name, expiration, etc. These IDs were kept in paper form, in the Data Center Manager's office. Of course, you had to jump thru the flaming hoops of Change Management first! BobL -Original Message- From: IBM Mainframe Discussion List [mailto:IBM-MAIN@LISTSERV.UA.EDU] On Behalf Of Charles Mills Sent: Tuesday, November 22, 2016 2:25 PM To: IBM-MAIN@LISTSERV.UA.EDU Subject: Re: Mainframe systems programmer ID 'vaulting' [ EXTERNAL ] Isn't this a violation of PCI DSS? "10.1 Implement audit trails to link all access to system components to each individual user." Charles -Original Message- From: IBM Mainframe Discussion List [mailto:IBM-MAIN@LISTSERV.UA.EDU] On Behalf Of Bigendian Smalls Sent: Tuesday, November 22, 2016 7:37 PM To: IBM-MAIN@LISTSERV.UA.EDU Subject: Re: Mainframe systems programmer ID 'vaulting' This is something I hadn't heard much about, but a couple questions come to mind - for anyone who has thought about or implemented this: 1) If you have a pool of IDs, then are you losing granularity with which you might want to assign privelages? Meaning you now have to have IDs that have exactly the same permissions - if they are user-agnostic (among some class of users obviously, like DEVs or SYSPROGs or whatever) - Seems like that is back to the old, "Create a new id. What perms to give him? Dunno, just build it like Chad's, they have the same job." Which has kind of gone out of style for obvious reasons (though still prevelant in practice). -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN This e-mail transmission may contain information that is proprietary, privileged and/or confidential and is intended exclusively for the person(s) to whom it is addressed. Any use, copying, retention or disclosure by any person other than the intended recipient or
Re: Mainframe systems programmer ID 'vaulting'
NTAC:3NS-20 We have a FIRECALL process now, but it's automated. The vaulting process has taken the place of this FIRECALL process for most applications teams. The systems programmers don't need to get a FIRECALL ID to update SYS1 datasets. We only need them when working with datasets that we don't ordinarily have access to, such as the datasets that come in a service pack that have HLQs that we don't have dataset rules for. We are currently vaulting a pool of logonids with god-like power for systems programmers to check out. When this is complete we will retire our old FIRECALL process. It's the next step, where our daily use TSO IDs may be vaulted, that scares us. Jim -Original Message- From: IBM Mainframe Discussion List [mailto:IBM-MAIN@LISTSERV.UA.EDU] On Behalf Of Lester, Bob Sent: Tuesday, November 22, 2016 4:04 PM To: IBM-MAIN@LISTSERV.UA.EDU Subject: Re: Mainframe systems programmer ID 'vaulting' For a while, about 15 years ago, we had "firecall" IDs. When you logged in, it prompted you for information that, in turn, updated RACF with Name, expiration, etc. These IDs were kept in paper form, in the Data Center Manager's office. Of course, you had to jump thru the flaming hoops of Change Management first! BobL -Original Message- From: IBM Mainframe Discussion List [mailto:IBM-MAIN@LISTSERV.UA.EDU] On Behalf Of Charles Mills Sent: Tuesday, November 22, 2016 2:25 PM To: IBM-MAIN@LISTSERV.UA.EDU Subject: Re: Mainframe systems programmer ID 'vaulting' [ EXTERNAL ] Isn't this a violation of PCI DSS? "10.1 Implement audit trails to link all access to system components to each individual user." Charles -Original Message- From: IBM Mainframe Discussion List [mailto:IBM-MAIN@LISTSERV.UA.EDU] On Behalf Of Bigendian Smalls Sent: Tuesday, November 22, 2016 7:37 PM To: IBM-MAIN@LISTSERV.UA.EDU Subject: Re: Mainframe systems programmer ID 'vaulting' This is something I hadn't heard much about, but a couple questions come to mind - for anyone who has thought about or implemented this: 1) If you have a pool of IDs, then are you losing granularity with which you might want to assign privelages? Meaning you now have to have IDs that have exactly the same permissions - if they are user-agnostic (among some class of users obviously, like DEVs or SYSPROGs or whatever) - Seems like that is back to the old, "Create a new id. What perms to give him? Dunno, just build it like Chad's, they have the same job." Which has kind of gone out of style for obvious reasons (though still prevelant in practice). -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN This e-mail transmission may contain information that is proprietary, privileged and/or confidential and is intended exclusively for the person(s) to whom it is addressed. Any use, copying, retention or disclosure by any person other than the intended recipient or the intended recipient's designees is strictly prohibited. If you are not the intended recipient or their designee, please notify the sender immediately by return e-mail and delete all copies. OppenheimerFunds may, at its sole discretion, monitor, review, retain and/or disclose the content of all email communications. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Mainframe systems programmer ID 'vaulting'
NTAC:3NS-20 Our company is undergoing a project to 'protect privileged access' by using a password vaulting product. We have been doing this for quite some time for applications teams who require higher levels of access to production datasets for problem resolution, installs, etc. The way it works is that a pool of logonids is created, along with an AD group that allows the appropriate applications folks to be able to 'check out' one of these pooled logonids for 24 hours via a web interface. The web interface uses the users lan password plus their secure key passcode and phrase to validate their identity. The project has now included Windows and Unix server admins, but instead of a pooled logonid these users have separate logonids with admin access and they 'check out' their own individual administrator logonid. Now the project has moved into the mainframe systems programmer space. So far we have used the 'privileges' on the logonid records as defined by our security product to limit this vaulting. Users with 'security' access must check out logonids from the vault. Users with the non-cncl privilege are next. During project discussions it has been brought up that the systems programmers, with their access to SYS1 datasets and operator commands, are privileged users by nature, and that eventually they are going to want to vault this access. We (the systems programmers) are strongly against this. It looks like at some point we will lose our battle and our access to the mainframe will be vaulted, meaning my entire team will need to check a logonid out of the password vault every morning before starting work. Our main argument now is that we do not want these logonids to be generic, pooled logonids, we want them to be basically the same as our own logonids so that we can see who did what by using the mainframe's built in logging (SMF data, ISPF stats, etc...). My questions are, are other companies using password vaulting or other multi-level authentication for mainframe systems programmer access? What else could we use in our argument against using generic, pooled logonids? Thanks in advance! Jim -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Ping times with shared OSA
NTAC:4UC-11 While investigating an issue, we noticed some variation in ping times from an RHEL 6 (on z/VM 6.2) to a z/OS 1.13 LPAR on the same CEC, using a shared 10G OSA. Times for 1000 pings: 1000 packets transmitted, 1000 received, 0% packet loss, time 999056ms rtt min/avg/max/mdev = 0.267/0.635/12.184/0.813 ms Anyone else see similar? Traceroute seems normal: traceroute to (IP addesses redacted) 30 hops max, 60 byte packets 1 (redacted) 0.710 ms 0.703 ms 0.723 ms 2 (redacted) 0.667 ms 0.657 ms 0.647 ms What are other people seeing? Thanks, Jim -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
