Marcy, I'll ask the vendor on the memory question...that's what I keep thinking also is that something must be running that spikes the memory and kills the machine before it has any time to recapture memory.
Kraig and Aaron, I'm also going to pass on the hardware information and see if that can lead the vendor down the right path as well. Thanks! Joell Chockley System Capacity/Performance Specialist Blue Cross Blue Shield of KS 1133 Topeka Blvd Topeka, KS 66629-0001 Work (785)291-7837 From: Marcy Cortes <marcy.d.cor...@wellsfargo.com> To: LINUX-390@vm.marist.edu Date: 12/07/2010 02:01 PM Subject: Re: Kernel panic - not syncing error on Red Hat Linux Well, that microcode fun was ours and what Linux got out of that was a "low address protection" error. This might be a clue: "Out of memory and no killable processes error" Do you need more memory? Got a memory leak?? Marcy -----Original Message----- From: Linux on 390 Port [mailto:linux-...@vm.marist.edu] On Behalf Of Graves, Aaron Sent: Tuesday, December 07, 2010 11:55 AM To: LINUX-390@vm.marist.edu Subject: Re: [LINUX-390] Kernel panic - not syncing error on Red Hat Linux Take a look at your hardware microcode level. This may be your problem. 060310 HIPER MCL's for 2097, 2098 Document number: 060310 Machine type: 2097, 2098 Models affected: All Models Abstract: HIPER MCLs have been released for machines at Driver 76 and Driver 79 Description HIPER MCLs have been released to address an issue reported in the field. This issue has only been seen in the Linux on System z operating system running as guest under zVM and has resulted in a Linux abend. This issue occurred when the physical CPU transitioned from handling one guest configuration to another. A very small window exists where the update of the hardware controls related to multiprocessor address translation were not handled properly. There are no methods to avoid exposure to this issue other than installing the specified HIPER MCL's. This problem has only been reported when running Linux on System z as a guest under zVM. However, IBM believes it could also occur when running other operating systems. Although an undetected data corruption has never been reported as a result of this issue, there is a very small chance that this could occur. The following HIPER MCLs have been released to eliminate the exposure: Driver 73 -- Driver-73 entered "End of MCL Support" August 2009 and a fix for this issue will not be released for this Driver. Product Engineering strongly recommends upgrading to Driver-79 to obtain this fix. Driver 76 -- MCL's N10967.101 and N10967.102 of the I390ML stream released to RETAIN on June 1st, 2010 in MCL bundle #47. Driver 79 -- MCL's N24406.032 and N24406.033 of the I390ML stream released to RETAIN on June 1st, 2010 in MCL bundle #31. Recommended action Product Engineering recommends the installation of HIPER MCLs . These MCLs can be installed concurrent with system operation. Note- Driver-76 is currently in "Limited MCL Support" mode and will enter "End of MCL Support" on June 30, 2010. Product Engineering strongly recommends upgrading to Driver-79 during you next microcode maintenance window. Driver-79 was released in November 2009 and is running on 40% of the z10 machines in the field. Driver-79 is also the last planned driver for z10. EDM Considerations: The installation of these HIPER MCL's at Driver-76 will exceed the current EDM min/max definition for concurrently upgrading to Driver-79. A new EDM sync point is being developed and is targeted to be released mid to late June, 2010. The following options are available: 1) Upgrade to Driver-79 and install the latest HIPER MCL's 2) Install the latest Driver-76 HIPER MCL's and wait for the next EDM sync point to be released. Note: Since these Driver-76 HIPER MCL's exceed the current EDM min/max level you need to take additional actions to ensure the MCL install and activate process overrides the current min/max level. To do this you need to insure that you select the 'check box' "Include internal code changes which will inhibit Concurrent Upgrade Engineering Changes (EC) task from being used to apply the next Licensed Internal Code (EC) level." before installing the Driver-76 HIPER MCL's. Regards, Aaron -----Original Message----- From: Linux on 390 Port [mailto:linux-...@vm.marist.edu] On Behalf Of Joell Chockley Sent: Tuesday, December 07, 2010 2:08 PM To: LINUX-390@VM.MARIST.EDU Subject: Kernel panic - not syncing error on Red Hat Linux Has anyone had any experience with the following error in Red Hat Linux? Kernel panic - not syncing: Out of memory and no killable processes error and system is frozen and has to be hard rebooted. cat /etc/redhat-release output is: Red Hat Enterprise Linux ES release 4 (Nahant Update 4). uname -a output is: Linux timmon1.corp.bcbsks.com 2.6.9-42.ELsmp #1 SMP Wed Jul 12 23:27:17 EDT 2006 i686 i686 i386 GNU/Linux I have a screen print of the console when it's frozen, but didn't know about attaching a graphic to a question. I also have a meminfo.out file and sysreport output that I can send if anyone thinks that would be helpful. We've changed the setting for lower_zone_protection as recommended from Red Hat and it's not having any impact on this problem occuring. Setting change from Red Hat: echo 100 > /proc/sys/vm/lower_zone_protection We've had this problem occur off and on with both of our Red Hat systems, but this one in particular has been having a major problem the last week and has been locking up about 3 times a day. We use the Linux systems with our CA Wily product for monitoring on the network. The top command right now shows this output: top - 12:44:49 up 2:16, 1 user, load average: 0.42, 0.21, 0.17 Tasks: 73 total, 1 running, 72 sleeping, 0 stopped, 0 zombie Cpu(s): 23.2% us, 0.3% sy, 0.0% ni, 64.6% id, 11.8% wa, 0.0% hi, 0.0% si Mem: 4144696k total, 1137308k used, 3007388k free, 448k buffers% Swap: 4088532k total, 0k used, 4088532k free, 41672k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 4256 root 16 0 470m 204m 9204 S 94 5.1 36:01.64 tim 3271 apache 16 0 13164 7068 3052 S 0 0.2 0:01.41 httpd And the system just locked up on me again...3rd time today. Any thoughts or ideas of where to look would be appreciated. Thanks! Joell Chockley System Capacity/Performance Specialist Blue Cross Blue Shield of KS 1133 Topeka Blvd Topeka, KS 66629-0001 Work (785)291-7837 CONFIDENTIALITY NOTICE: This email message and any attachments are for the sole use of the intended recipient(s) and may contain proprietary, confidential, trade secret or privileged information. Any unauthorized review use, disclosure or distribution is prohibited and may be a violation of law. If you are not the intended recipient or a person responsible for delivering this message to an intended recipient, please contact the sender by reply email and destroy all copies of the original message. ---------------------------------------------------------------------- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 ---------------------------------------------------------------------- For more information on Linux on System z, visit http://wiki.linuxvm.org/ ---------------------------------------------------------------------- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 ---------------------------------------------------------------------- For more information on Linux on System z, visit http://wiki.linuxvm.org/ ---------------------------------------------------------------------- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 ---------------------------------------------------------------------- For more information on Linux on System z, visit http://wiki.linuxvm.org/ CONFIDENTIALITY NOTICE: This email message and any attachments are for the sole use of the intended recipient(s) and may contain proprietary, confidential, trade secret or privileged information. Any unauthorized review use, disclosure or distribution is prohibited and may be a violation of law. If you are not the intended recipient or a person responsible for delivering this message to an intended recipient, please contact the sender by reply email and destroy all copies of the original message. ---------------------------------------------------------------------- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 ---------------------------------------------------------------------- For more information on Linux on System z, visit http://wiki.linuxvm.org/
