Re: Server rebooting after doing a CP Q DA from the console
On Sat, 19 Mar 2011 23:10:16 -0300 Mauro Souza thoriu...@gmail.com wrote: I saw this problem some time ago with an Oracle RAC guest. It haven't set the CP SET RUN ON, and as soon as the client issued some #CP Q SOMETHING, the server froze down, and linux rebooted. Looks like Oracle RAC have some kind of watchdog, and as CP MODE stops running Linux kernel for a little moment, the watchdog thinks the system froze down, and reboots the system. Setting RUN ON solved the problem. You can try this, it won't hurt, and I think RUN ON should be the default. There is the important hint: if you have Oracle RAC and the watchdog is running the z/VM guest may not stop for longer periods of time. My guts feeling is that the large output of the #CP Q xyz command stopped the linux guest for too long. Once the output completed the guest continued and the Oracle watchdog did what it is programmed to do: reboot. -- blue skies, Martin. Reality continues to ruin my life. - Calvin. -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/
Re: Server rebooting after doing a CP Q DA from the console
Chen, No, console remained connected, the screen goes from displaying the DASD to a Linux boot. I was in such shock as it happened; I failed to capture a screen shot. Our terminal settings were/are: LINEND # BRKKEY PA1 Alan, Martin, Mauro Rob Van der Heij, Thanks very much for your feedback, I think what you added to this discussion is the solution. All of you give great value to this List Serv. This being a production system, I cannot test it right way out of the risk of job security, but I am in the process of setting up a test server attached to a large number of unused DASD devices and see if I can recreate the problem and then test your solution. I plan to test it with and without an ORACLE RAC. James Chaplin Systems Programmer, MVS, zVM zLinux -Original Message- From: Linux on 390 Port [mailto:LINUX-390@VM.MARIST.EDU] On Behalf Of Eddie Chen Sent: Friday, March 18, 2011 5:10 PM To: LINUX-390@VM.MARIST.EDU Subject: Re: Server rebooting after doing a CP Q DA from the console After you issue the cp query, did you disconnect? -Original Message- Mauro Souza thoriu...@gmail.com wrote: I saw this problem some time ago with an Oracle RAC guest. It haven't set the CP SET RUN ON, and as soon as the client issued some #CP Q SOMETHING, the server froze down, and linux rebooted. Looks like Oracle RAC have some kind of watchdog, and as CP MODE stops running Linux kernel for a little moment, the watchdog thinks the system froze down, and reboots the system. Setting RUN ON solved the problem. You can try this, it won't hurt, and I think RUN ON should be the default. Martin Schwidefsky schwidef...@de.ibm.com wrote: There is the important hint: if you have Oracle RAC and the watchdog is running the z/VM guest may not stop for longer periods of time. My guts feeling is that the large output of the #CP Q xyz command stopped the linux guest for too long. Once the output completed the guest continued and the Oracle watchdog did what it is programmed to do: reboot. Alan Altmark alan_altm...@us.ibm.com wrote: Regardless, when output comes to the console, there is no buffering, even if it thousands of lines long. That's ok since the output doesn't go through the virtual machine OS, but directly from CP to the 3270. Only the 3270 PA1 key (the default break key) will stop the output. CP QUERY TERMINAL to look at the BRKKEY setting. Pressing the break key will cause CP to put up a CP READ, canceling any pending output. If #cp had in fact been caught by CP, not Linux, there would have been no buffering, no error message, and no server abend. Ergo, #cp was not caught by CP. Rob van der Heij rvdh...@gmail.com wrote: So we're talking about virtual-MP in both cases? Are both also using cpuplugd to vary off CPUs maybe? Setting RUN ON will only prevent the CP READ upon reconnect. You should not need to reconnect a perfectly runing system. With RUN ON you may get buried in console output that prevents you from doing what you came for. I recommend to have RUN OFF and be aware of the CP READ when you have to reconnect. -Original Message- From: Linux on 390 Port [mailto:LINUX-390@VM.MARIST.EDU] On Behalf Of CHAPLIN, JAMES (CTR) Sent: Friday, March 18, 2011 5:07 PM To: LINUX-390@VM.MARIST.EDU Subject: Server rebooting after doing a CP Q DA from the console Found a problem today with CP QUERY command that caused our Linux Server to reboot. Using vmcp from a zLinux server, I issued the following command: vmcp q dasd | less Binary file (standard input) matches Error: output (21282 bytes) was truncated, try --buffer to increase size This guest has a large number of DASD attached to it (Oracle database) causing this problem. (Should have used the 'vmcp -b q dasd | less') But I thought I would try the same command from the guest console (#CP Q DA), and got the following response just before the server rebooted: DASD 4886 ON DASD 4886 R/W VI1304 SUBCHANNEL = 014C DASD 4887 ON DASD 4887 R/W VI1305 SUBCHANNEL = 014D Error: output (21282 bytes) was truncated, Two questions as I am a VM rookie, How to you modify the buffer from the console for the CP command? How do you display the buffer size available for a CP command? Also: Why would a query command on a console bring down a linux guest, but vmcp query does not? Is this a know bug? James Chaplin Systems Programmer, MVS, zVM zLinux -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/ Please consider the environment before printing this email. Visit our website at http://www.nyse.com Note
Re: Server rebooting after doing a CP Q DA from the console
On Sun, Mar 20, 2011 at 3:10 AM, Mauro Souza thoriu...@gmail.com wrote: I saw this problem some time ago with an Oracle RAC guest. It haven't set the CP SET RUN ON, and as soon as the client issued some #CP Q SOMETHING, the server froze down, and linux rebooted. Looks like Oracle RAC have some So we're talking about virtual-MP in both cases? Are both also using cpuplugd to vary off CPUs maybe? kind of watchdog, and as CP MODE stops running Linux kernel for a little moment, the watchdog thinks the system froze down, and reboots the system. Setting RUN ON solved the problem. You can try this, it won't hurt, and I think RUN ON should be the default. Setting RUN ON will only prevent the CP READ upon reconnect. You should not need to reconnect a perfectly runing system. With RUN ON you may get buried in console output that prevents you from doing what you came for. I recommend to have RUN OFF and be aware of the CP READ when you have to reconnect. Rob -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/
Re: Server rebooting after doing a CP Q DA from the console
On Saturday, 03/19/2011 at 02:24 EDT, CHAPLIN, JAMES (CTR) james.chap...@associates.dhs.gov wrote: Yes I understand Alan is right on, and that vmcp protect's itself with a limited buffer. But when I did the same command from the console instead of using the vmcp command on the guest, it brought the entire server down. VMCP limits the amount of output through buffers, how does VM limit output from commands from the console? I should have phrased my question that way. Sorry I wasn't clear, James. When you enter #CP some command on the virtual machine console, two things have be true for CP to see it as a command: 1. CP has to be looking for line editing characters. That is, CP SET LINEDIT ON has to be in effect, AND 2. CP has to be looking for # as the end-of-line editing character. CP TERMINAL LINEND #. QUERY TERMINAL will tell you what the linend character is, and QUERY SET will tell you if CP even cares. Regardless, when output comes to the console, there is no buffering, even if it thousands of lines long. That's ok since the output doesn't go through the virtual machine OS, but directly from CP to the 3270. Only the 3270 PA1 key (the default break key) will stop the output. CP QUERY TERMINAL to look at the BRKKEY setting. Pressing the break key will cause CP to put up a CP READ, canceling any pending output. If #cp had in fact been caught by CP, not Linux, there would have been no buffering, no error message, and no server abend. Ergo, #cp was not caught by CP. Alan Altmark z/VM and Linux on System z Consultant IBM System Lab Services and Training ibm.com/systems/services/labservices office: 607.429.3323 mobile; 607.321.7556 alan_altm...@us.ibm.com IBM Endicott -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/
Re: Server rebooting after doing a CP Q DA from the console
James, As Alan said, someone is intercepting in Linux your CP command. The vmcp q term does not generate so much output that it does not fit in the standard vmcp buffer, so you should be fairly safe issueing that command. An other option is to issue from an other authorized VM user (for instance MAINT) the command FOR linux_guest_id CMD QUERY TERMINAL For what it is worth, in my days as an operator (a long long time ago), we intercepted the D T command, sometimes issued by MVS operators to Display Time. Only on VM it was a Dump Terminal, so we were less amused when that got issued under OPERATOR .. Ronald van der Laan -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/
Re: Server rebooting after doing a CP Q DA from the console
Yes I understand Alan is right on, and that vmcp protect's itself with a limited buffer. But when I did the same command from the console instead of using the vmcp command on the guest, it brought the entire server down. VMCP limits the amount of output through buffers, how does VM limit output from commands from the console? I should have phrased my question that way. James Chaplin Systems Programmer, MVS, zVM zLinux -Original Message- From: Linux on 390 Port [mailto:LINUX-390@VM.MARIST.EDU] On Behalf Of Ronald van der Laan Sent: Saturday, March 19, 2011 9:23 AM To: LINUX-390@VM.MARIST.EDU Subject: Re: Server rebooting after doing a CP Q DA from the console James, As Alan said, someone is intercepting in Linux your CP command. The vmcp q term does not generate so much output that it does not fit in the standard vmcp buffer, so you should be fairly safe issueing that command. An other option is to issue from an other authorized VM user (for instance MAINT) the command FOR linux_guest_id CMD QUERY TERMINAL For what it is worth, in my days as an operator (a long long time ago), we intercepted the D T command, sometimes issued by MVS operators to Display Time. Only on VM it was a Dump Terminal, so we were less amused when that got issued under OPERATOR .. Ronald van der Laan -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/ -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/
Re: Server rebooting after doing a CP Q DA from the console
CP doesn't do that. In real time you can use the PA1 key to stop the output of a CP command. But CP doesn't impose buffer limits. A true pound signCP command will not limit the output buffer. It looks like somehow the terminal line end was changed and Linux was involved in output swallowing. Unless the server restart was not related! as obscure as that seems. What about a temporary CP READ state by using PA1 key, then issuing the cp command? Your buffer should not be limited and the output is not given to Linux. David Original Message Subject: Re: Server rebooting after doing a CP Q DA from the console From: CHAPLIN, JAMES (CTR) james.chap...@associates.dhs.gov Date: Sat, March 19, 2011 2:23 pm To: LINUX-390@VM.MARIST.EDU Yes I understand Alan is right on, and that vmcp protect's itself with a limited buffer. But when I did the same command from the console instead of using the vmcp command on the guest, it brought the entire server down. VMCP limits the amount of output through buffers, how does VM limit output from commands from the console? I should have phrased my question that way. James Chaplin Systems Programmer, MVS, zVM zLinux -Original Message- From: Linux on 390 Port [mailto:LINUX-390@VM.MARIST.EDU] On Behalf Of Ronald van der Laan Sent: Saturday, March 19, 2011 9:23 AM To: LINUX-390@VM.MARIST.EDU Subject: Re: Server rebooting after doing a CP Q DA from the console James, As Alan said, someone is intercepting in Linux your CP command. The vmcp q term does not generate so much output that it does not fit in the standard vmcp buffer, so you should be fairly safe issueing that command. An other option is to issue from an other authorized VM user (for instance MAINT) the command FOR linux_guest_id CMD QUERY TERMINAL For what it is worth, in my days as an operator (a long long time ago), we intercepted the D T command, sometimes issued by MVS operators to Display Time. Only on VM it was a Dump Terminal, so we were less amused when that got issued under OPERATOR .. Ronald van der Laan -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/ -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/ -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/
Re: Server rebooting after doing a CP Q DA from the console
On 3/19/2011 at 02:34 PM, David Kreuter dkreu...@vm-resources.com wrote: It looks like somehow the terminal line end was changed and Linux was involved in output swallowing. Unless the server restart was not related! as obscure as that seems. Which a #cp q term will verify, with a very small amount of output that won't endanger the system. The keyword to be looking for is LINEND on the first line of output. If it doesn't say LINEND # then that's the problem. Mark Post -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/
Re: Server rebooting after doing a CP Q DA from the console
After you issue the cp query, did you disconnect? -Original Message- From: Linux on 390 Port [mailto:LINUX-390@VM.MARIST.EDU] On Behalf Of CHAPLIN, JAMES (CTR) Sent: Friday, March 18, 2011 5:07 PM To: LINUX-390@VM.MARIST.EDU Subject: Server rebooting after doing a CP Q DA from the console Found a problem today with CP QUERY command that caused our Linux Server to reboot. Using vmcp from a zLinux server, I issued the following command: vmcp q dasd | less Binary file (standard input) matches Error: output (21282 bytes) was truncated, try --buffer to increase size This guest has a large number of DASD attached to it (Oracle database) causing this problem. (Should have used the 'vmcp -b q dasd | less') But I thought I would try the same command from the guest console (#CP Q DA), and got the following response just before the server rebooted: DASD 4886 ON DASD 4886 R/W VI1304 SUBCHANNEL = 014C DASD 4887 ON DASD 4887 R/W VI1305 SUBCHANNEL = 014D Error: output (21282 bytes) was truncated, Two questions as I am a VM rookie, How to you modify the buffer from the console for the CP command? How do you display the buffer size available for a CP command? Also: Why would a query command on a console bring down a linux guest, but vmcp query does not? Is this a know bug? James Chaplin Systems Programmer, MVS, zVM zLinux -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/ Please consider the environment before printing this email. Visit our website at http://www.nyse.com Note: The information contained in this message and any attachment to it is privileged, confidential and protected from disclosure. If the reader of this message is not the intended recipient, or an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately by replying to the message, and please delete it from your system. Thank you. NYSE Euronext. -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/
Re: Server rebooting after doing a CP Q DA from the console
I saw this problem some time ago with an Oracle RAC guest. It haven't set the CP SET RUN ON, and as soon as the client issued some #CP Q SOMETHING, the server froze down, and linux rebooted. Looks like Oracle RAC have some kind of watchdog, and as CP MODE stops running Linux kernel for a little moment, the watchdog thinks the system froze down, and reboots the system. Setting RUN ON solved the problem. You can try this, it won't hurt, and I think RUN ON should be the default. Mauro http://mauro.limeiratem.com - registered Linux User: 294521 Scripture is both history, and a love letter from God. On Fri, Mar 18, 2011 at 6:10 PM, Eddie Chen ec...@nyx.com wrote: After you issue the cp query, did you disconnect? -Original Message- From: Linux on 390 Port [mailto:LINUX-390@VM.MARIST.EDU] On Behalf Of CHAPLIN, JAMES (CTR) Sent: Friday, March 18, 2011 5:07 PM To: LINUX-390@VM.MARIST.EDU Subject: Server rebooting after doing a CP Q DA from the console Found a problem today with CP QUERY command that caused our Linux Server to reboot. Using vmcp from a zLinux server, I issued the following command: vmcp q dasd | less Binary file (standard input) matches Error: output (21282 bytes) was truncated, try --buffer to increase size This guest has a large number of DASD attached to it (Oracle database) causing this problem. (Should have used the 'vmcp -b q dasd | less') But I thought I would try the same command from the guest console (#CP Q DA), and got the following response just before the server rebooted: DASD 4886 ON DASD 4886 R/W VI1304 SUBCHANNEL = 014C DASD 4887 ON DASD 4887 R/W VI1305 SUBCHANNEL = 014D Error: output (21282 bytes) was truncated, Two questions as I am a VM rookie, How to you modify the buffer from the console for the CP command? How do you display the buffer size available for a CP command? Also: Why would a query command on a console bring down a linux guest, but vmcp query does not? Is this a know bug? James Chaplin Systems Programmer, MVS, zVM zLinux -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/ Please consider the environment before printing this email. Visit our website at http://www.nyse.com Note: The information contained in this message and any attachment to it is privileged, confidential and protected from disclosure. If the reader of this message is not the intended recipient, or an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately by replying to the message, and please delete it from your system. Thank you. NYSE Euronext. -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/ -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/
Re: Server rebooting after doing a CP Q DA from the console
On Friday, 03/18/2011 at 05:07 EDT, CHAPLIN, JAMES (CTR) james.chap...@associates.dhs.gov wrote: But I thought I would try the same command from the guest console (#CP Q DA), and got the following response just before the server rebooted: DASD 4886 ON DASD 4886 R/W VI1304 SUBCHANNEL = 014C DASD 4887 ON DASD 4887 R/W VI1305 SUBCHANNEL = 014D Error: output (21282 bytes) was truncated, Two questions as I am a VM rookie, How to you modify the buffer from the console for the CP command? How do you display the buffer size available for a CP command? You don't. The Error: output you see wasn't generated by CP; it came from Linux. Ergo, the command you issued didn't actually go to CP. It wouldn't surprise me if someone set up a shell script of some sort to turn #cp (a habit) into vmcp. Why would a query command on a console bring down a linux guest, but vmcp query does not? Is this a know bug? Try again, but issue vmcp query term before you try the #cp. I often see that the LINEND character has been set to something other than # in order to allow use of # in Linux. Alan Altmark z/VM and Linux on System z Consultant IBM System Lab Services and Training ibm.com/systems/services/labservices office: 607.429.3323 mobile; 607.321.7556 alan_altm...@us.ibm.com IBM Endicott -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/
Re: Server rebooting after doing a CP Q DA from the console
My biggest problem is that I have no way to recreate the problem without impacting our production. The vmcp command is at a level of protection against this problem. But what we want to understand is why would a simple #CP Q DA bring down the guest linux? You should have seen my face as I watched this Linux guest go into a reboot from the console view on a Friday evening :-( I was unable to capture the true output when the CP Q DA failed. James Chaplin Systems Programmer, MVS, zVM zLinux -Original Message- From: Linux on 390 Port [mailto:LINUX-390@VM.MARIST.EDU] On Behalf Of Alan Altmark Sent: Friday, March 18, 2011 5:54 PM To: LINUX-390@VM.MARIST.EDU Subject: Re: Server rebooting after doing a CP Q DA from the console On Friday, 03/18/2011 at 05:07 EDT, CHAPLIN, JAMES (CTR) james.chap...@associates.dhs.gov wrote: But I thought I would try the same command from the guest console (#CP Q DA), and got the following response just before the server rebooted: DASD 4886 ON DASD 4886 R/W VI1304 SUBCHANNEL = 014C DASD 4887 ON DASD 4887 R/W VI1305 SUBCHANNEL = 014D Error: output (21282 bytes) was truncated, Two questions as I am a VM rookie, How to you modify the buffer from the console for the CP command? How do you display the buffer size available for a CP command? You don't. The Error: output you see wasn't generated by CP; it came from Linux. Ergo, the command you issued didn't actually go to CP. It wouldn't surprise me if someone set up a shell script of some sort to turn #cp (a habit) into vmcp. Why would a query command on a console bring down a linux guest, but vmcp query does not? Is this a know bug? Try again, but issue vmcp query term before you try the #cp. I often see that the LINEND character has been set to something other than # in order to allow use of # in Linux. Alan Altmark z/VM and Linux on System z Consultant IBM System Lab Services and Training ibm.com/systems/services/labservices office: 607.429.3323 mobile; 607.321.7556 alan_altm...@us.ibm.com IBM Endicott -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/ -- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For more information on Linux on System z, visit http://wiki.linuxvm.org/
