Re: DASD Moderate Alert from EMC DMX4
I think it is time I clarified a few things. Our EMC Symmetrix is supported by our storage group while I support our mainframe. There was some miscommunication at the start. The storage group looked at what error messages they were getting and were told that these are only informational messages and could be ignored. Because of that the storage group did not think it was worth the effort to open an incident. So when I thought we had talked with EMC and were not getting support we had not even opened an incident call. Since I thought we should open an incident the storage group guru and I got together and it was decided I should open an incident. Since then we have had great support. EMC support called in and found out the problem is that the "VDI" data pool is at 91% full and the Symmetrix has been trying to phone home, but there was some problem and the calls were failing. They have sent out a CE to fix both problems. So from now on when we see a problem we will in! itiate an incident with EMC and the storage group will help us resolve it I also want to thank everyone for the help here. Using the information provided here we were going in the right direction and were close to resolving the problem ourselves. So thanks again for all the help. Brad Wissink Information Technology Services Iowa State University 515-294-3088 -Original Message- From: IBM Mainframe Discussion List [mailto:ibm-m...@bama.ua.edu] On Behalf Of R.S. Sent: Tuesday, August 31, 2010 1:58 PM To: IBM-MAIN@bama.ua.edu Subject: Re: DASD Moderate Alert from EMC DMX4 W dniu 2010-08-31 18:23, Pinnacle pisze: > - Original Message - From: "R.S." > Newsgroups: bit.listserv.ibm-main > Sent: Tuesday, August 31, 2010 4:02 AM > Subject: Re: DASD Moderate Alert from EMC DMX4 > > >> Pinnacle pisze: >> [...] >>>>>>> Brad Wissink 8/30/2010 2:53 PM >>> >>>> We started getting the following message from our EMC DMX4 Symmetrix >>>> over >>>> the weekend. We are talking with EMC but not getting very far. So, does >>>> anyone know what the message is telling us? >>>> >>>> *IEA480E 400D,DASD,MODERATE ALERT,MT=2105,SER=0507-03966, 259 >>>> REFCODE=E467-0471-,VOLSER=DB0022,ID=01,SENSE=1B10 >>>> 0D268F01 >>>> 8F44 7114 07000F7E 0001E467 05100200 FE00 >>>> >>> >>> Brad, >>> >>> Scott's right. There is no excuse for EMC not to know EXACTLY what >>> this message means. I would have serious doubts about any DASD vendor >>> who could not decipher this message on their own hardware. Ridiculous. >> >> How do you know that your assumption is true? "We were talking with >> EMC" could mean everything. Not every EMC employee is CE, not every >> talk is problem submission, not every "not getting very far" means >> that EMC technician doesn't know the explanation or didn't presented >> it to the customer. >> Disclaimer: I'm not neither EMC, nor customer attorney. >> > > I took the OP at his word. Why? Did the statement sound absolutely unambigously (clear-cut)? Actually I have no reason to disbelieve him, but either no reason to believe every word or further assume that some formal actions happened. It's my humble opinion. And general rule of thumb, much less humble: DON'T BELIEVE. If someone tells you that IEFBR14 is abending, then DON'T BELIEVE HIM. CHECK EVERY INFORMATION PROVIDED. CHECK AGAIN. (here is the place to put any fun story from computer support, like "my application doesn't work, I can't check anything, cause it's dark, we have power failure"). > Bottom line is that someone at EMC should > know the EXACT meaning of this error message. I bet that SOMEONE (not necessarily everyone) knows it. > It is usually incumbent on > the vendor to put the customer in touch with that person, not incumbent > on the customer to mount a wumpus hunt for that person within EMC. Well, NO. I don't know what communication between customer na vendor did happen actually. So, I can talk generally: NO, it's customer responsibility to inform vendor in AGREED WAY that something happened. Informal communication may or may not cause any reaction. Last, but not least: lack of EMC explanation on this forum proves ABSOLUTELY NOTHING. This is not the place where they should discuss customers problems. My €0.02 -- Radoslaw Skorupka Lodz, Poland -- BRE Bank SA ul. Senatorska 18 00-950 Warszawa www.brebank.pl Sd Rejonowy dla m. st. Warszawy XII Wydzia Gospodarczy Krajowego Rejestru Sdowego, nr rejestru przedsibiorców KRS 025237 NIP: 526-021-50-88 Wedug
Re: DASD Moderate Alert from EMC DMX4
W dniu 2010-08-31 18:23, Pinnacle pisze: - Original Message - From: "R.S." Newsgroups: bit.listserv.ibm-main Sent: Tuesday, August 31, 2010 4:02 AM Subject: Re: DASD Moderate Alert from EMC DMX4 Pinnacle pisze: [...] Brad Wissink 8/30/2010 2:53 PM >>> We started getting the following message from our EMC DMX4 Symmetrix over the weekend. We are talking with EMC but not getting very far. So, does anyone know what the message is telling us? *IEA480E 400D,DASD,MODERATE ALERT,MT=2105,SER=0507-03966, 259 REFCODE=E467-0471-,VOLSER=DB0022,ID=01,SENSE=1B10 0D268F01 8F44 7114 07000F7E 0001E467 05100200 FE00 Brad, Scott's right. There is no excuse for EMC not to know EXACTLY what this message means. I would have serious doubts about any DASD vendor who could not decipher this message on their own hardware. Ridiculous. How do you know that your assumption is true? "We were talking with EMC" could mean everything. Not every EMC employee is CE, not every talk is problem submission, not every "not getting very far" means that EMC technician doesn't know the explanation or didn't presented it to the customer. Disclaimer: I'm not neither EMC, nor customer attorney. I took the OP at his word. Why? Did the statement sound absolutely unambigously (clear-cut)? Actually I have no reason to disbelieve him, but either no reason to believe every word or further assume that some formal actions happened. It's my humble opinion. And general rule of thumb, much less humble: DON'T BELIEVE. If someone tells you that IEFBR14 is abending, then DON'T BELIEVE HIM. CHECK EVERY INFORMATION PROVIDED. CHECK AGAIN. (here is the place to put any fun story from computer support, like "my application doesn't work, I can't check anything, cause it's dark, we have power failure"). Bottom line is that someone at EMC should know the EXACT meaning of this error message. I bet that SOMEONE (not necessarily everyone) knows it. It is usually incumbent on the vendor to put the customer in touch with that person, not incumbent on the customer to mount a wumpus hunt for that person within EMC. Well, NO. I don't know what communication between customer na vendor did happen actually. So, I can talk generally: NO, it's customer responsibility to inform vendor in AGREED WAY that something happened. Informal communication may or may not cause any reaction. Last, but not least: lack of EMC explanation on this forum proves ABSOLUTELY NOTHING. This is not the place where they should discuss customers problems. My €0.02 -- Radoslaw Skorupka Lodz, Poland -- BRE Bank SA ul. Senatorska 18 00-950 Warszawa www.brebank.pl Sd Rejonowy dla m. st. Warszawy XII Wydzia Gospodarczy Krajowego Rejestru Sdowego, nr rejestru przedsibiorców KRS 025237 NIP: 526-021-50-88 Wedug stanu na dzie 01.01.2009 r. kapita zakadowy BRE Banku SA (w caoci wpacony) wynosi 118.763.528 zotych. W zwizku z realizacj warunkowego podwyszenia kapitau zakadowego, na podstawie uchway XXI WZ z dnia 16 marca 2008r., oraz uchway XVI NWZ z dnia 27 padziernika 2008r., moe ulec podwyszeniu do kwoty 123.763.528 z. Akcje w podwyszonym kapitale zakadowym BRE Banku SA bd w caoci opacone. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: DASD Moderate Alert from EMC DMX4
>EMC told VITA that the outage was “unprecedented” Precedent: A certain bank, in Toronto, had a similar failure many years ago (mid-1990's). Albeit, it was older technology, but the primary and secondary failed. - I'm a SuperHero with neither powers, nor motivation! Kimota! -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: DASD Moderate Alert from EMC DMX4
- Original Message - From: "R.S." Newsgroups: bit.listserv.ibm-main Sent: Tuesday, August 31, 2010 4:02 AM Subject: Re: DASD Moderate Alert from EMC DMX4 Pinnacle pisze: [...] Brad Wissink 8/30/2010 2:53 PM >>> We started getting the following message from our EMC DMX4 Symmetrix over the weekend. We are talking with EMC but not getting very far. So, does anyone know what the message is telling us? *IEA480E 400D,DASD,MODERATE ALERT,MT=2105,SER=0507-03966, 259 REFCODE=E467-0471-,VOLSER=DB0022,ID=01,SENSE=1B10 0D268F01 8F44 7114 07000F7E 0001E467 05100200 FE00 Brad, Scott's right. There is no excuse for EMC not to know EXACTLY what this message means. I would have serious doubts about any DASD vendor who could not decipher this message on their own hardware. Ridiculous. How do you know that your assumption is true? "We were talking with EMC" could mean everything. Not every EMC employee is CE, not every talk is problem submission, not every "not getting very far" means that EMC technician doesn't know the explanation or didn't presented it to the customer. Disclaimer: I'm not neither EMC, nor customer attorney. I took the OP at his word. Bottom line is that someone at EMC should know the EXACT meaning of this error message. It is usually incumbent on the vendor to put the customer in touch with that person, not incumbent on the customer to mount a wumpus hunt for that person within EMC. Regards, Tom Conley -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: DASD Moderate Alert from EMC DMX4
Lucky it's not a DMX3, apparently. :) ‘Unprecedented’ storage failure disrupts US state 31 August 2010 One-in-a-billion’ SAN glitch followed by redundant system failure affects a third of Virginia's government agencies Citizens in the US state of Virginia were unable to access a number of public services after the state government’s storage area network (SAN) suffered an ‘unprecedented’ hardware failure. The SAN, a DMX-3 from storage infrastructure vendor EMC, failed last Wednesday afternoon, causing 228 of the Virginia Information Technologies Agency’s (VITA) servers to crash. Critically, VITA’s redundant systems – the spare infrastructure designed to take the strain when primary systems fail – also malfunctioned. Interesting Links IBM takes blame for Singapore bank outage Big Blue concedes that employee’s “procedural error” was responsible for seven-hour outage after Asian bank points finger This outage disrupted services for 27 state agencies, including the Department of Motor Vehicles (DMV), Department of Taxation and the State Board of Elections. Most of the services were back online by yesterday morning but the DMV is still unable to issue driver’s licenses. EMC told VITA that the outage was “unprecedented”, according to a statement from Virginia’s secretary of technology Jim Duffey. “The manufacturer reports that the system and its underlying technology have an exemplary history of reliability, industry-leading data availability of more than 99.999 percent and no similar failure has occurred” VITA is now “working tirelessly” to restore lost data, Duffey said. Earlier, this year US email hosting provider Intermedia suffered an outage after a hardware failure in its EMC storage area network. “This failure caused the entire load for that SAN to be shifted to the service processor on the redundant controller node,” the company said at the time. “The spare capacity on the single service processor was not enough to handle the entire load of all systems connected to the SAN.” -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: DASD Moderate Alert from EMC DMX4
Radoslaw has a good point. On the other hand, EMC has people who monitor this server. Someone at EMC could/should have jumped on this several days ago. Any vendor that does NOT monitor this server and respond quickly to potential bad publicity is acting foolishly, IMHO. Caveat: I assumed that the OP was correct in saying that the vendor involved is EMC and in reproducing the error messages. Bill Fairchild Rocket Software -Original Message- From: IBM Mainframe Discussion List [mailto:ibm-m...@bama.ua.edu] On Behalf Of R.S. Sent: Tuesday, August 31, 2010 2:59 AM To: IBM-MAIN@bama.ua.edu Subject: Re: DASD Moderate Alert from EMC DMX4 Pinnacle pisze: >>>>> Brad Wissink 8/30/2010 2:53 PM >>> >> We started getting the following message from our EMC DMX4 Symmetrix over >> the weekend. We are talking with EMC but not getting very far. So, >> does >> anyone know what the message is telling us? >> >> *IEA480E 400D,DASD,MODERATE ALERT,MT=2105,SER=0507-03966, 259 >> REFCODE=E467-0471-,VOLSER=DB0022,ID=01,SENSE=1B10 >> 0D268F01 >> 8F44 7114 07000F7E 0001E467 05100200 FE00 >> > > Brad, > > Scott's right. There is no excuse for EMC not to know EXACTLY what this > message means. I would have serious doubts about any DASD vendor who > could not decipher this message on their own hardware. Ridiculous. How do you know that your assumption is true? "We were talking with EMC" could mean everything. Not every EMC employee is CE, not every talk is problem submission, not every "not getting very far" means that EMC technician doesn't know the explanation or didn't presented it to the customer. Disclaimer: I'm not neither EMC, nor customer attorney. -- Radoslaw Skorupka -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: DASD Moderate Alert from EMC DMX4
Pinnacle pisze: [...] Brad Wissink 8/30/2010 2:53 PM >>> We started getting the following message from our EMC DMX4 Symmetrix over the weekend. We are talking with EMC but not getting very far. So, does anyone know what the message is telling us? *IEA480E 400D,DASD,MODERATE ALERT,MT=2105,SER=0507-03966, 259 REFCODE=E467-0471-,VOLSER=DB0022,ID=01,SENSE=1B10 0D268F01 8F44 7114 07000F7E 0001E467 05100200 FE00 Brad, Scott's right. There is no excuse for EMC not to know EXACTLY what this message means. I would have serious doubts about any DASD vendor who could not decipher this message on their own hardware. Ridiculous. How do you know that your assumption is true? "We were talking with EMC" could mean everything. Not every EMC employee is CE, not every talk is problem submission, not every "not getting very far" means that EMC technician doesn't know the explanation or didn't presented it to the customer. Disclaimer: I'm not neither EMC, nor customer attorney. -- Radoslaw Skorupka Lodz, Poland -- BRE Bank SA ul. Senatorska 18 00-950 Warszawa www.brebank.pl Sd Rejonowy dla m. st. Warszawy XII Wydzia Gospodarczy Krajowego Rejestru Sdowego, nr rejestru przedsibiorców KRS 025237 NIP: 526-021-50-88 Wedug stanu na dzie 01.01.2009 r. kapita zakadowy BRE Banku SA (w caoci wpacony) wynosi 118.763.528 zotych. W zwizku z realizacj warunkowego podwyszenia kapitau zakadowego, na podstawie uchway XXI WZ z dnia 16 marca 2008r., oraz uchway XVI NWZ z dnia 27 padziernika 2008r., moe ulec podwyszeniu do kwoty 123.763.528 z. Akcje w podwyszonym kapitale zakadowym BRE Banku SA bd w caoci opacone. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: DASD Moderate Alert from EMC DMX4
There is a Open Systems Solutions Enabler command called SYMEVENT that allows one to view events that occurred in the (maybe more detail) SYMM\DMX\VMAX. The command is documented in the Solutions Enabler Command Line Reference HTML which can be downloaded from EMC Powerlink. Don't know if the mainframe has that capability or a comparable command. You might also be able to retrieve this event if you have EMC ControlCenter installed . This output from this command may provide more detail? If Can you access the SYMAPI database you may also be able to view its contents for more information. If your SYMM\DMX\VMAX is also shared with an Open Systems Host than run the SYMEVENT and SYMAUDIT commands from there. These events are stored in the Symm\DMX\VMAX in SFS volumes. -Original Message- From: IBM Mainframe Discussion List [mailto:ibm-m...@bama.ua.edu] On Behalf Of Lizette Koehler Sent: Monday, August 30, 2010 5:53 PM To: IBM-MAIN@bama.ua.edu Subject: Re: DASD Moderate Alert from EMC DMX4 \ > We started getting the following message from our EMC DMX4 Symmetrix > over > the weekend. We are talking with EMC but not getting very far. So, > does > anyone know what the message is telling us? > > *IEA480E 400D,DASD,MODERATE ALERT,MT=2105,SER=0507-03966, 259 > REFCODE=E467-0471-,VOLSER=DB0022,ID=01,SENSE=1B10 > 0D268F01 > 8F44 7114 07000F7E 0001E467 05100200 FE00 > Brad, Look up the REF Code in the Resource Pak. Or ask your EMC CE what a E467 is. I cannot remember which one means unable to phone home. Also note, that EMC can only report on the last DASD that IOS knew about, so it may not even be that particular drive. I have EMC and I always open a case to EMC support on these types of issues. Lizette -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: DASD Moderate Alert from EMC DMX4
- Original Message - From: "Scott Rowe" Newsgroups: bit.listserv.ibm-main Sent: Monday, August 30, 2010 3:07 PM Subject: Re: DASD Moderate Alert from EMC DMX4 If EMC is having a problem understanding that message, then I would suggest it is time to replace your EMC hardware with a product from a company that knows how to spell mainframe properly. Brad Wissink 8/30/2010 2:53 PM >>> We started getting the following message from our EMC DMX4 Symmetrix over the weekend. We are talking with EMC but not getting very far. So, does anyone know what the message is telling us? *IEA480E 400D,DASD,MODERATE ALERT,MT=2105,SER=0507-03966, 259 REFCODE=E467-0471-,VOLSER=DB0022,ID=01,SENSE=1B10 0D268F01 8F44 7114 07000F7E 0001E467 05100200 FE00 Brad, Scott's right. There is no excuse for EMC not to know EXACTLY what this message means. I would have serious doubts about any DASD vendor who could not decipher this message on their own hardware. Ridiculous. Regards, Tom Conley -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: DASD Moderate Alert from EMC DMX4
\ > We started getting the following message from our EMC DMX4 Symmetrix > over > the weekend. We are talking with EMC but not getting very far. So, > does > anyone know what the message is telling us? > > *IEA480E 400D,DASD,MODERATE ALERT,MT=2105,SER=0507-03966, 259 > REFCODE=E467-0471-,VOLSER=DB0022,ID=01,SENSE=1B10 > 0D268F01 > 8F44 7114 07000F7E 0001E467 05100200 FE00 > Brad, Look up the REF Code in the Resource Pak. Or ask your EMC CE what a E467 is. I cannot remember which one means unable to phone home. Also note, that EMC can only report on the last DASD that IOS knew about, so it may not even be that particular drive. I have EMC and I always open a case to EMC support on these types of issues. Lizette -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: DASD Moderate Alert from EMC DMX4
Brad, We moved off EMC recently, however, I recall this type of error being associated with FICON microcode levels (IC3). The DMX4 may need updates. Perhaps recent updates added to your processor. -Original Message- From: IBM Mainframe Discussion List [mailto:ibm-m...@bama.ua.edu] On Behalf Of Brad Wissink Sent: Monday, August 30, 2010 2:54 PM To: IBM-MAIN@bama.ua.edu Subject: DASD Moderate Alert from EMC DMX4 We started getting the following message from our EMC DMX4 Symmetrix over the weekend. We are talking with EMC but not getting very far. So, does anyone know what the message is telling us? *IEA480E 400D,DASD,MODERATE ALERT,MT=2105,SER=0507-03966, 259 REFCODE=E467-0471-,VOLSER=DB0022,ID=01,SENSE=1B10 0D268F01 8F44 7114 07000F7E 0001E467 05100200 FE00 Thanks -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html This e-mail message and any attachments transmitted with it are confidential and are intended solely for the use of its authorized recipient(s). If you are not an intended or authorized recipient, you are hereby notified that any disclosure, copying, distribution or taking any action in reliance on the information contained in this e-mail is prohibited. If you have received this message in error or are not authorized to receive it, please immediately notify the sender and delete the original message and all copies of it from your computer. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: DASD Moderate Alert from EMC DMX4
On Mon, 30 Aug 2010 13:53:38 -0500, Brad Wissink wrote: >We started getting the following message from our EMC DMX4 Symmetrix over >the weekend. We are talking with EMC but not getting very far. So, does >anyone know what the message is telling us? > >*IEA480E 400D,DASD,MODERATE ALERT,MT=2105,SER=0507-03966, 259 > REFCODE=E467-0471-,VOLSER=DB0022,ID=01,SENSE=1B10 >0D268F01 > 8F44 7114 07000F7E 0001E467 05100200 FE00 > >Thanks > An E467 refcode in EMC world means an RDF error occurred across the link from the other box. Sent Brad the EMC refcodes cheat sheet off list. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: DASD Moderate Alert from EMC DMX4
If EMC is having a problem understanding that message, then I would suggest it is time to replace your EMC hardware with a product from a company that knows how to spell mainframe properly. >>> Brad Wissink 8/30/2010 2:53 PM >>> We started getting the following message from our EMC DMX4 Symmetrix over the weekend. We are talking with EMC but not getting very far. So, does anyone know what the message is telling us? *IEA480E 400D,DASD,MODERATE ALERT,MT=2105,SER=0507-03966, 259 REFCODE=E467-0471-,VOLSER=DB0022,ID=01,SENSE=1B10 0D268F01 8F44 7114 07000F7E 0001E467 05100200 FE00 Thanks -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html CONFIDENTIALITY/EMAIL NOTICE: The material in this transmission contains confidential and privileged information intended only for the addressee. If you are not the intended recipient, please be advised that you have received this material in error and that any forwarding, copying, printing, distribution, use or disclosure of the material is strictly prohibited. If you have received this material in error, please (i) do not read it, (ii) reply to the sender that you received the message in error, and (iii) erase or destroy the material. Emails are not secure and can be intercepted, amended, lost or destroyed, or contain viruses. You are deemed to have accepted these risks if you communicate with us by email. Thank you. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
