Hi Rex, You might want to protect QUIESCE and a few other similar commands so that they can only be done at a system console and not through SDSF and the like. See article "Protect Shutdown Commands" in our RSH RACF Newsletter:
https://www.rshconsulting.com/racftips/RSH_Consulting__RACF_Tips__January_2013.pdf Regards, Bob Robert S. Hansel 2021 #IBMChampion Lead RACF Specialist RSH Consulting, Inc. 617-969-8211 www.linkedin.com/in/roberthansel www.twitter.com/RSH_RACF www.rshconsulting.com --------------------------------------------------------------------------- Upcoming RSH RACF Training - WebEx - RACF Audit & Compliance Roadmap - OCT 18-22, 2021 - RACF Level I Administration - APR 12-16, 2021 - RACF Level II Administration - NOV 15-19, 2021 - RACF Level III Admin, Audit, & Compliance - NOV 1-5, 2021 - RACF - Securing z/OS UNIX - SEPT 20-24, 2021 --------------------------------------------------------------------------- -----Original Message----- Date: Wed, 24 Mar 2021 17:50:07 +0000 From: "Pommier, Rex" <rpomm...@sfgmembers.com> Subject: Re: [External] Re: z14 HMC log information Hi Radoslaw, I knew you meant it as a joke and I took it as such. Hence my smiley face. The OPERCMDS class has several entries in it but somehow QUIESCE was missed from way back when for a specific lock down so it was allowed by a more generic profile. I checked the type80 records and there was nothing immediately before the quiesce command was entered. O well, we figured out what happened and put security in place to minimize the possibility of it happening again, and we now know what to do if it does happen so we can get the system back up without issue and be able to find and train the guilty party. Rex -----Original Message----- From: IBM Mainframe Discussion List <IBM-MAIN@LISTSERV.UA.EDU> On Behalf Of Radoslaw Skorupka Sent: Wednesday, March 24, 2021 10:26 AM To: IBM-MAIN@LISTSERV.UA.EDU Subject: [External] Re: z14 HMC log information Obviously the part about killing was only poor joke, but there is some sense hidden in. I mean it is good idea to talk to person who did it. Not to punish, but talk and explain and hear his/her explanations. For RACF admin it is quite obvious the security model should be somehow checked. Again the person can have good explanation of current state of protection. Regarding traces - It is a little bit hard to test, especially without access to mainframe ;-) but I guess SMF80 can be written just before system freeze. Note it is RACF security check - it happens BEFORE the command is interpreted by the system. Simpler example: when you issue CANCEL CICSABC and you don't have such started task, you first will be checked by RACF (and maybe rejected) and then the command is really issued, and you will get answer like "there is no such started task to cancel". BTW: I imagined what would happen after such case on production... -- Radoslaw Skorupka (looking for new job) Lodz, Poland W dniu 24.03.2021 o 14:58, Pommier, Rex pisze: > I'm going to agree with *most* of it. I don't like the part about killing > the RACF admin. I'm not the one who initially set up the OPERCMDS security > but I missed the fact the QUIESCE command wasn't set as "don't let anybody > use". Hari-kari is not on my bucket list. :-) > > On to Radoslaw's comment about logging - it is logged, after the fact. > QUIESCE does exactly that - it stops the LPAR in its tracks. Do not pass Go, > do not collect $200. No z/OS logging at the time it happens. IBM hardware > support found and reported the wait state back to us from some hardware logs > that were forwarded to them from our CE. The z/OS logging takes place after > the PSW restart from the HMC occurs and yes, it shows the console or user > that executed the command. However in our case since the LPAR stopped in the > middle of the day and we had managers breathing down our necks to get the > system back up we didn't have time to properly diagnose until after the fact > - which included an IPL which in turn did not allow the logging of the > quiesce command to take place. > > Rex > > -----Original Message----- > From: IBM Mainframe Discussion List <IBM-MAIN@LISTSERV.UA.EDU> On Behalf Of > Carmen Vitullo > Sent: Wednesday, March 24, 2021 8:07 AM > To: IBM-MAIN@LISTSERV.UA.EDU > Subject: [External] Re: z14 HMC log information > > agree 100%, when I tested the command on my sandbox system I see my ID in the > syslog as the culprit :) if done from a console, then the console name is > shown. > > Carmen Vitullo > > > > -----Original Message----- > > From: Radoslaw <r.skoru...@hotmail.com> > To: IBM-MAIN <IBM-MAIN@LISTSERV.UA.EDU> > Date: Wednesday, 24 March 2021 8:01 AM CDT > Subject: Re: z14 HMC log information > > IMHO there should be a trace in a syslog. Maybe that part of syslog is > somehow lost. > And it would be good idea to have SMF record for that command. I don't know > about console commands, but SMF80 could be cut if you take care about > AUDIT(ALL(READ)) in advance. I mean RACF profile. > > Then find the person and kill him. Don't forget to kill RACF admin who forgot > to protect the command. Kill'em all. ;-) > > Seriously: it is a symptom of bad security model. > > > -- > Radoslaw Skorupka > (looking for new job) > Lodz, Poland > > > > W dniu 24.03.2021 o 13:54, Pommier, Rex pisze: >> Hi Carmen, >> >> We use the quiesce parameter of the reset command as well. I'm thinking >> that's what happened here as well, but we have no idea "whodunit". We had 1 >> operator on duty that day and he was off doing something else when it >> happened so the command made it into the system via an SDSF-like console >> session. >> >> Rex >> >> -----Original Message----- >> From: IBM Mainframe Discussion List <IBM-MAIN@LISTSERV.UA.EDU> On >> Behalf Of Carmen Vitullo >> Sent: Wednesday, March 24, 2021 7:43 AM >> To: IBM-MAIN@LISTSERV.UA.EDU >> Subject: [External] Re: z14 HMC log information >> >> There is a queisce command to put a job to sleep, and I think that's what my >> operator was thinking he was doing, for a runaway job we do use automation >> to queisce a job and send us an alert. >> from SDSF there's a quiesce line command RQ it translates to RESET >> jobname,QUIESCE >> >> >> Carmen Vitullo >> >> >> >> -----Original Message----- >> >> From: Radoslaw <r.skoru...@hotmail.com> >> To: IBM-MAIN <IBM-MAIN@LISTSERV.UA.EDU> >> Date: Wednesday, 24 March 2021 5:35 AM CDT >> Subject: Re: z14 HMC log information >> >> IMHO not really. >> >> By freezing the system you disrupt all the online processing and you loose >> capability to find out what's wrong. >> I had to do with CPU hogging tasks and spool 100% filled by some ugly job. >> In both cases the very first thing to fix it was to logon to the system. >> >> -- >> Radoslaw Skorupka >> (looking for new job) >> Lodz, Poland >> >> >> >> W dniu 24.03.2021 o 03:30, kekronbekron pisze: >>> It's useful when you want to say "STOP IT!" to run-away CPU or spool jobs. >>> Will buy you some time to investigate. >>> >>> - KB >>> >>> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ >>> On Tuesday, March 23, 2021 7:07 PM, Pommier, Rex <rpomm...@sfgmembers.com> >>> wrote: >>> >>>> Hey Carmen, >>>> >>>> At least you had an operator tell you what they did. :-) Once we >>>> determined what had happened (after a painful middle-of-the-day IPL) the >>>> only response I got back from anybody was "I didn't do anything like >>>> that!" But now that I know what that command does and how it works I doubt >>>> I'll ever forget this one. >>>> >>>> Rex >>>> >>>> -----Original Message----- >>>> From: IBM Mainframe Discussion List IBM-MAIN@LISTSERV.UA.EDU On >>>> Behalf Of Carmen Vitullo >>>> Sent: Tuesday, March 23, 2021 7:47 AM >>>> To: IBM-MAIN@LISTSERV.UA.EDU >>>> Subject: Re: [External] Re: z14 HMC log information >>>> >>>> WOW! this takes me back 20 years when an operator wanted to quiesce >>>> a job and issues that command from the console he called my told me >>>> what he did, I researched what that command does, seemed to me it >>>> was like on the old system consoles O2 and O3 IIRC one was a PSW >>>> stop and one a restart, I found the restart just for grins and it >>>> worked, been a long time since I've hear tell someone had that same >>>> experience great find >>>> >>>> Carmen Vitullo >>>> >>>> -----Original Message----- >>>> >>>> From: Rex rpomm...@sfgmembers.com >>>> To: IBM-MAIN IBM-MAIN@LISTSERV.UA.EDU >>>> Date: Monday, 22 March 2021 5:18 PM CDT >>>> Subject: Re: [External] Re: z14 HMC log information >>>> >>>> OK, this is embarrassing. We found the problem - but don't know the >>>> culprit. We discovered the quiesce command wasn't locked down and >>>> somebody somewhere keyed it in. The reason we don't know who/where >>>> is because our SDSF-like product had the capability to also do a >>>> quiesce and since we didn't know that's what caused it, we IPLed the >>>> LPAR. I did some research after the fact to determine just how >>>> quiesce works (been working on these things for 30 years and never >>>> had a reason to even play with quiesce). Now I know - and I know how >>>> to make the machine go again, picking up where it stopped. PSW >>>> Restart from the HMC doesn't do what I consider a "restart" but a >>>> "take the brakes off and let it run". :-) >>>> >>>> IBM hardware support found the wait state and informed us of it. >>>> >>>> Thanks everybody for the bandwidth and suggestions. >>>> >>>> Rex >>>> >>>> -----Original Message----- >>>> From: IBM Mainframe Discussion List IBM-MAIN@LISTSERV.UA.EDU On >>>> Behalf Of kekronbekron >>>> Sent: Wednesday, March 17, 2021 8:50 AM >>>> To: IBM-MAIN@LISTSERV.UA.EDU >>>> Subject: [External] Re: z14 HMC log information >>>> >>>> Please do share when you find out! >>>> >>>> I wonder if an LPAR with full BCPii authority over the box can silently >>>> query/log information from the HMC for monitoring, i.e., actions occuring >>>> not via BCPii itself, but just accessing HMC logs. >>>> >>>> - KB >>>> >>>> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ >>>> On Wednesday, March 17, 2021 6:17 PM, Carmen Vitullo cvitu...@hughes.net >>>> wrote: >>>> >>>> >>>>> so the only other thing I can think of is if you are in a sysplex and >>>>> your SFM policy took the system out of the plex, XCF may have not seen a >>>>> heartbeat in a long enough time it partitioned the system from the plex, >>>>> I've had this happen before, you should see some IXC messages if this >>>>> happened. >>>>> when this happened to me there was no indication anyone did >>>>> anything the system was just removed from the plex >>>>> >>>>> Carmen Vitullo >>>>> -----Original Message----- >>>>> From: Rex rpomm...@sfgmembers.com >>>>> To: IBM-MAIN IBM-MAIN@LISTSERV.UA.EDU >>>>> Date: Tuesday, 16 March 2021 4:33 PM CDT >>>>> Subject: Re: z14 HMC log information Nothing out of the ordinary in >>>>> the HMC logs. >>>>> -----Original Message----- >>>>> From: IBM Mainframe Discussion List IBM-MAIN@LISTSERV.UA.EDU On >>>>> Behalf Of Carmen Vitullo >>>>> Sent: Tuesday, March 16, 2021 1:10 PM >>>>> To: IBM-MAIN@LISTSERV.UA.EDU >>>>> Subject: [External] Re: z14 HMC log information so it looks like >>>>> one place to check is from the HMC, /console actions/View control >>>>> tasks performed, this may get you what you need another place is >>>>> from console actions, View console events >>>>> >>>>> Carmen Vitullo >>>>> -----Original Message----- >>>>> From: Rex rpomm...@sfgmembers.com >>>>> To: IBM-MAIN IBM-MAIN@LISTSERV.UA.EDU >>>>> Date: Tuesday, 16 March 2021 12:51 PM CDT >>>>> Subject: Re: z14 HMC log information I found the log - nothing in >>>>> it. Heading off to IBM. >>>>> Rex >>>>> -----Original Message----- >>>>> From: IBM Mainframe Discussion List IBM-MAIN@LISTSERV.UA.EDU On >>>>> Behalf Of Pommier, Rex >>>>> Sent: Tuesday, March 16, 2021 12:24 PM >>>>> To: IBM-MAIN@LISTSERV.UA.EDU >>>>> Subject: [External] z14 HMC log information Hi all, Probably a >>>>> simple question but I'm not having any luck looking for it so I'm asking >>>>> here. >>>>> Is there a centralized place on a z14 HMC to show activity occurring, >>>>> LPAR activation/deactivation, pretty much anything that happens on an HMC >>>>> or on a z. We just took a hit where one of our LPARs and we're not seeing >>>>> anything. No hardware messages, no dumps, no nothing. The syslog on the >>>>> LPAR shows everything running as normal then it just stopped, 15 minutes >>>>> later are the IPL starting messages. It almost looks like somebody hit >>>>> the "deactivate" button on the HMC so I'm looking for any log info from a >>>>> hardware point of view. >>>>> Thanks, >>>>> Rex >>>>> >> ---------------------------------------------------------------------- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
