z/VM RedHat Virtual Machine Memory abend
Hi s390x Gurus I´m new to Linux on z and we are conducting a POC test of Linux Red Hat with Oracle running under z/VM 5.4 RSU 1003 and a z/10 BC model A00 (IFL Only). If linux virtual machine is defined with 64G in directory we got an abend during startup (see below). Doing some tests I found that I can define it until 59G in directory (60794036k from TOP linux command) and it will startup correctly. My questions are: 1 - Is this a limitation of linux s390x or Red Hat distribution? 2 - Is this a linux s390x bug or Red Hat distribution? 3 - How can I circunvent this so we can continue with POC (Proof Of Concept)? Thanks in advance for all hints and tips about. 00: Booting default (2.6.18-194.8.1.el5)... Linux version 2.6.18-194.8.1.el5 (mockbu...@s390-002.build.bos.redhat.com) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-48)) #1 SMP Wed Jun 23 11:09:16 EDT 2010 We are running under VM (64 bit mode) Detected 3 CPU's Boot cpu address 0 Built 1 zonelists. Total pages: 16777216 Kernel command line: root=LABEL=/ BOOT_IMAGE=0 PID hash table entries: 4096 (order: 12, 32768 bytes) Dentry cache hash table entries: 8388608 (order: 14, 67108864 bytes) Inode-cache hash table entries: 4194304 (order: 13, 33554432 bytes) *Memory: 65951564k/67108864k available* (3414k kernel code, 0k reserved, 2256k dat a, 136k init) Write protected kernel read-only data: 0x356000 - 0x409fff Calibrating delay loop... 2981.88 BogoMIPS (lpj=14909440) Security Framework v1.0.0 initialized SELinux: Initializing. selinux_register_security: Registering secondary module capability Capability LSM initialized as secondary Mount-cache hash table entries: 256 cpu 0 phys_idx=0 vers=FF ident=01F2D6 machine=2098 unused=8000 cpu 1 phys_idx=1 vers=FF ident=01F2D6 machine=2098 unused=8000 cpu 2 phys_idx=2 vers=FF ident=01F2D6 machine=2098 unused=8000 Brought up 3 CPUs migration_cost=1000 checking if image is initramfs...it isn't (bad gzip magic numbers); looks like a n initrd Freeing initrd memory: 2759k freed list_del corruption. prev-next should be 8001bff0, but was 8001 c840 kernel BUG at lib/list_debug.c:65! illegal operation: 0001 Ý#1¨ CPU: 0 Not tainted 2.6.18-194.8.1.el5 #1 Process swapper (pid: 1, task: c611f788, ksp: c6123b10) Krnl PSW : 040400018000 002685d6 (list_del+0x9e/0xcc) Krnl GPRS: 0015 c6123cb8 0026 0400 0012a83e 0733 000a 0050e400 0001 8001bf90 8001bff0 8001bfce 00369878 002685d2 c6123d10 Krnl Code: d2 07 10 08 20 08 e3 10 d0 08 00 04 e3 10 20 08 00 24 e3 10 Call Trace: (Ý002685d2¨ list_del+0x9a/0xcc) Ý0017820a¨ free_pages_bulk+0x9e/0x34c Ý0017868a¨ free_hot_cold_page+0x1d2/0x1f4 Ý001009c0¨ free_initrd_mem+0x8c/0xc8 Ý005944ee¨ free_initrd+0xba/0xe4 Ý0058e324¨ init+0x1a0/0x32c Ý0010558e¨ kernel_thread_starter+0x6/0xc Ý00105588¨ kernel_thread_starter+0x0/0xc 0Kernel panic - not syncing: Fatal exception: panic_on_oops 01: HCPGSP2629I The virtual machine is placed in CP mode due to a SIGP stop from CPU 00. 02: HCPGSP2629I The virtual machine is placed in CP mode due to a SIGP stop from CPU 00. 00: HCPGIR450W CP entered; disabled wait PSW 00020001 8000 0010BDBE -- Carlos Bodra IBM Certified Specialist System z Sao Paulo - Brazil
Re: z/VM RedHat Virtual Machine Memory abend
Are you Running VM on this or zLinux only on an LPAR? Larry Davis From: The IBM z/VM Operating System [mailto:IBMVM@LISTSERV.UARK.EDU] On Behalf Of Carlos Bodra - Pessoal Sent: Wednesday, April 20, 2011 6:19 PM To: IBMVM@LISTSERV.UARK.EDU Subject: z/VM RedHat Virtual Machine Memory abend Hi s390x Gurus I´m new to Linux on z and we are conducting a POC test of Linux Red Hat with Oracle running under z/VM 5.4 RSU 1003 and a z/10 BC model A00 (IFL Only). If linux virtual machine is defined with 64G in directory we got an abend during startup (see below). Doing some tests I found that I can define it until 59G in directory (60794036k from TOP linux command) and it will startup correctly. My questions are: 1 - Is this a limitation of linux s390x or Red Hat distribution? 2 - Is this a linux s390x bug or Red Hat distribution? 3 - How can I circunvent this so we can continue with POC (Proof Of Concept)? Thanks in advance for all hints and tips about. 00: Booting default (2.6.18-194.8.1.el5)... Linux version 2.6.18-194.8.1.el5 (mockbu...@s390-002.build.bos.redhat.commailto:mockbu...@s390-002.build.bos.redhat.com) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-48)) #1 SMP Wed Jun 23 11:09:16 EDT 2010 We are running under VM (64 bit mode) Detected 3 CPU's Boot cpu address 0 Built 1 zonelists. Total pages: 16777216 Kernel command line: root=LABEL=/ BOOT_IMAGE=0 PID hash table entries: 4096 (order: 12, 32768 bytes) Dentry cache hash table entries: 8388608 (order: 14, 67108864 bytes) Inode-cache hash table entries: 4194304 (order: 13, 33554432 bytes) Memory: 65951564k/67108864k available (3414k kernel code, 0k reserved, 2256k dat a, 136k init) Write protected kernel read-only data: 0x356000 - 0x409fff Calibrating delay loop... 2981.88 BogoMIPS (lpj=14909440) Security Framework v1.0.0 initialized SELinux: Initializing. selinux_register_security: Registering secondary module capability Capability LSM initialized as secondary Mount-cache hash table entries: 256 cpu 0 phys_idx=0 vers=FF ident=01F2D6 machine=2098 unused=8000 cpu 1 phys_idx=1 vers=FF ident=01F2D6 machine=2098 unused=8000 cpu 2 phys_idx=2 vers=FF ident=01F2D6 machine=2098 unused=8000 Brought up 3 CPUs migration_cost=1000 checking if image is initramfs...it isn't (bad gzip magic numbers); looks like a n initrd Freeing initrd memory: 2759k freed list_del corruption. prev-next should be 8001bff0, but was 8001 c840 kernel BUG at lib/list_debug.c:65! illegal operation: 0001 Ý#1¨ CPU: 0 Not tainted 2.6.18-194.8.1.el5 #1 Process swapper (pid: 1, task: c611f788, ksp: c6123b10) Krnl PSW : 040400018000 002685d6 (list_del+0x9e/0xcc) Krnl GPRS: 0015 c6123cb8 0026 0400 0012a83e 0733 000a 0050e400 0001 8001bf90 8001bff0 8001bfce 00369878 002685d2 c6123d10 Krnl Code: d2 07 10 08 20 08 e3 10 d0 08 00 04 e3 10 20 08 00 24 e3 10 Call Trace: (Ý002685d2¨ list_del+0x9a/0xcc) Ý0017820a¨ free_pages_bulk+0x9e/0x34c Ý0017868a¨ free_hot_cold_page+0x1d2/0x1f4 Ý001009c0¨ free_initrd_mem+0x8c/0xc8 Ý005944ee¨ free_initrd+0xba/0xe4 Ý0058e324¨ init+0x1a0/0x32c Ý0010558e¨ kernel_thread_starter+0x6/0xc Ý00105588¨ kernel_thread_starter+0x0/0xc 0Kernel panic - not syncing: Fatal exception: panic_on_oops 01: HCPGSP2629I The virtual machine is placed in CP mode due to a SIGP stop from CPU 00. 02: HCPGSP2629I The virtual machine is placed in CP mode due to a SIGP stop from CPU 00. 00: HCPGIR450W CP entered; disabled wait PSW 00020001 8000 0010BDBE -- Carlos Bodra IBM Certified Specialist System z Sao Paulo - Brazil
Re: z/VM RedHat Virtual Machine Memory abend
Larry, We are running it z/VM 5.4 RSU 1003 (lastest), but problem is connected to Linux, because I changed MAINT user memory to 128GB and all runs fine. This machine has no CP´s just IFL´s processors. Carlos Bodra IBM Certified Specialist System z Sao Paulo - Brazil Em 20/04/2011 19:24, Davis, Larry (National VM/VSE Capability) escreveu: Are you Running VM on this or zLinux only on an LPAR? Larry Davis** *From:*The IBM z/VM Operating System [mailto:IBMVM@LISTSERV.UARK.EDU] *On Behalf Of *Carlos Bodra - Pessoal *Sent:* Wednesday, April 20, 2011 6:19 PM *To:* IBMVM@LISTSERV.UARK.EDU *Subject:* z/VM RedHat Virtual Machine Memory abend Hi s390x Gurus I´m new to Linux on z and we are conducting a POC test of Linux Red Hat with Oracle running under z/VM 5.4 RSU 1003 and a z/10 BC model A00 (IFL Only). If linux virtual machine is defined with 64G in directory we got an abend during startup (see below). Doing some tests I found that I can define it until 59G in directory (60794036k from TOP linux command) and it will startup correctly. My questions are: 1 - Is this a limitation of linux s390x or Red Hat distribution? 2 - Is this a linux s390x bug or Red Hat distribution? 3 - How can I circunvent this so we can continue with POC (Proof Of Concept)? Thanks in advance for all hints and tips about. 00: Booting default (2.6.18-194.8.1.el5)... Linux version 2.6.18-194.8.1.el5 (mockbu...@s390-002.build.bos.redhat.com mailto:mockbu...@s390-002.build.bos.redhat.com) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-48)) #1 SMP Wed Jun 23 11:09:16 EDT 2010 We are running under VM (64 bit mode) Detected 3 CPU's Boot cpu address 0 Built 1 zonelists. Total pages: 16777216 Kernel command line: root=LABEL=/ BOOT_IMAGE=0 PID hash table entries: 4096 (order: 12, 32768 bytes) Dentry cache hash table entries: 8388608 (order: 14, 67108864 bytes) Inode-cache hash table entries: 4194304 (order: 13, 33554432 bytes) *Memory: 65951564k/67108864k available*(3414k kernel code, 0k reserved, 2256k dat a, 136k init) Write protected kernel read-only data: 0x356000 - 0x409fff Calibrating delay loop... 2981.88 BogoMIPS (lpj=14909440) Security Framework v1.0.0 initialized SELinux: Initializing. selinux_register_security: Registering secondary module capability Capability LSM initialized as secondary Mount-cache hash table entries: 256 cpu 0 phys_idx=0 vers=FF ident=01F2D6 machine=2098 unused=8000 cpu 1 phys_idx=1 vers=FF ident=01F2D6 machine=2098 unused=8000 cpu 2 phys_idx=2 vers=FF ident=01F2D6 machine=2098 unused=8000 Brought up 3 CPUs migration_cost=1000 checking if image is initramfs...it isn't (bad gzip magic numbers); looks like a n initrd Freeing initrd memory: 2759k freed list_del corruption. prev-next should be 8001bff0, but was 8001 c840 kernel BUG at lib/list_debug.c:65! illegal operation: 0001 Ý#1¨ CPU: 0 Not tainted 2.6.18-194.8.1.el5 #1 Process swapper (pid: 1, task: c611f788, ksp: c6123b10) Krnl PSW : 040400018000 002685d6 (list_del+0x9e/0xcc) Krnl GPRS: 0015 c6123cb8 0026 0400 0012a83e 0733 000a 0050e400 0001 8001bf90 8001bff0 8001bfce 00369878 002685d2 c6123d10 Krnl Code: d2 07 10 08 20 08 e3 10 d0 08 00 04 e3 10 20 08 00 24 e3 10 Call Trace: (Ý002685d2¨ list_del+0x9a/0xcc) Ý0017820a¨ free_pages_bulk+0x9e/0x34c Ý0017868a¨ free_hot_cold_page+0x1d2/0x1f4 Ý001009c0¨ free_initrd_mem+0x8c/0xc8 Ý005944ee¨ free_initrd+0xba/0xe4 Ý0058e324¨ init+0x1a0/0x32c Ý0010558e¨ kernel_thread_starter+0x6/0xc Ý00105588¨ kernel_thread_starter+0x0/0xc 0Kernel panic - not syncing: Fatal exception: panic_on_oops 01: HCPGSP2629I The virtual machine is placed in CP mode due to a SIGP stop from CPU 00. 02: HCPGSP2629I The virtual machine is placed in CP mode due to a SIGP stop from CPU 00. 00: HCPGIR450W CP entered; disabled wait PSW 00020001 8000 0010BDBE -- Carlos Bodra IBM Certified Specialist System z Sao Paulo - Brazil
Re: z/VM RedHat Virtual Machine Memory abend
What is your Hardware Memory configuration for Main Storage and XSTORE for VM Larry Davis From: The IBM z/VM Operating System [mailto:IBMVM@LISTSERV.UARK.EDU] On Behalf Of Carlos Bodra - Pessoal Sent: Wednesday, April 20, 2011 6:39 PM To: IBMVM@LISTSERV.UARK.EDU Subject: Re: z/VM RedHat Virtual Machine Memory abend Larry, We are running it z/VM 5.4 RSU 1003 (lastest), but problem is connected to Linux, because I changed MAINT user memory to 128GB and all runs fine. This machine has no CP´s just IFL´s processors. Carlos Bodra IBM Certified Specialist System z Sao Paulo - Brazil Em 20/04/2011 19:24, Davis, Larry (National VM/VSE Capability) escreveu: Are you Running VM on this or zLinux only on an LPAR? Larry Davis From: The IBM z/VM Operating System [mailto:IBMVM@LISTSERV.UARK.EDU] On Behalf Of Carlos Bodra - Pessoal Sent: Wednesday, April 20, 2011 6:19 PM To: IBMVM@LISTSERV.UARK.EDUmailto:IBMVM@LISTSERV.UARK.EDU Subject: z/VM RedHat Virtual Machine Memory abend Hi s390x Gurus I´m new to Linux on z and we are conducting a POC test of Linux Red Hat with Oracle running under z/VM 5.4 RSU 1003 and a z/10 BC model A00 (IFL Only). If linux virtual machine is defined with 64G in directory we got an abend during startup (see below). Doing some tests I found that I can define it until 59G in directory (60794036k from TOP linux command) and it will startup correctly. My questions are: 1 - Is this a limitation of linux s390x or Red Hat distribution? 2 - Is this a linux s390x bug or Red Hat distribution? 3 - How can I circunvent this so we can continue with POC (Proof Of Concept)? Thanks in advance for all hints and tips about. 00: Booting default (2.6.18-194.8.1.el5)... Linux version 2.6.18-194.8.1.el5 (mockbu...@s390-002.build.bos.redhat.commailto:mockbu...@s390-002.build.bos.redhat.com) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-48)) #1 SMP Wed Jun 23 11:09:16 EDT 2010 We are running under VM (64 bit mode) Detected 3 CPU's Boot cpu address 0 Built 1 zonelists. Total pages: 16777216 Kernel command line: root=LABEL=/ BOOT_IMAGE=0 PID hash table entries: 4096 (order: 12, 32768 bytes) Dentry cache hash table entries: 8388608 (order: 14, 67108864 bytes) Inode-cache hash table entries: 4194304 (order: 13, 33554432 bytes) Memory: 65951564k/67108864k available (3414k kernel code, 0k reserved, 2256k dat a, 136k init) Write protected kernel read-only data: 0x356000 - 0x409fff Calibrating delay loop... 2981.88 BogoMIPS (lpj=14909440) Security Framework v1.0.0 initialized SELinux: Initializing. selinux_register_security: Registering secondary module capability Capability LSM initialized as secondary Mount-cache hash table entries: 256 cpu 0 phys_idx=0 vers=FF ident=01F2D6 machine=2098 unused=8000 cpu 1 phys_idx=1 vers=FF ident=01F2D6 machine=2098 unused=8000 cpu 2 phys_idx=2 vers=FF ident=01F2D6 machine=2098 unused=8000 Brought up 3 CPUs migration_cost=1000 checking if image is initramfs...it isn't (bad gzip magic numbers); looks like a n initrd Freeing initrd memory: 2759k freed list_del corruption. prev-next should be 8001bff0, but was 8001 c840 kernel BUG at lib/list_debug.c:65! illegal operation: 0001 Ý#1¨ CPU: 0 Not tainted 2.6.18-194.8.1.el5 #1 Process swapper (pid: 1, task: c611f788, ksp: c6123b10) Krnl PSW : 040400018000 002685d6 (list_del+0x9e/0xcc) Krnl GPRS: 0015 c6123cb8 0026 0400 0012a83e 0733 000a 0050e400 0001 8001bf90 8001bff0 8001bfce 00369878 002685d2 c6123d10 Krnl Code: d2 07 10 08 20 08 e3 10 d0 08 00 04 e3 10 20 08 00 24 e3 10 Call Trace: (Ý002685d2¨ list_del+0x9a/0xcc) Ý0017820a¨ free_pages_bulk+0x9e/0x34c Ý0017868a¨ free_hot_cold_page+0x1d2/0x1f4 Ý001009c0¨ free_initrd_mem+0x8c/0xc8 Ý005944ee¨ free_initrd+0xba/0xe4 Ý0058e324¨ init+0x1a0/0x32c Ý0010558e¨ kernel_thread_starter+0x6/0xc Ý00105588¨ kernel_thread_starter+0x0/0xc 0Kernel panic - not syncing: Fatal exception: panic_on_oops 01: HCPGSP2629I The virtual machine is placed in CP mode due to a SIGP stop from CPU 00. 02: HCPGSP2629I The virtual machine is placed in CP mode due to a SIGP stop from CPU 00. 00: HCPGIR450W CP entered; disabled wait PSW 00020001 8000 0010BDBE -- Carlos Bodra IBM Certified Specialist System z Sao Paulo - Brazil
Re: z/VM RedHat Virtual Machine Memory abend
q stor STORAGE = 48G CONFIGURED = 48G INC = 128M STANDBY = 0 RESERVED = 0 Ready; T=0.01/0.01 20:33:09 No xstore defined. Carlos Bodra IBM Certified Specialist System z Sao Paulo - Brazil Em 20/04/2011 20:04, Davis, Larry (National VM/VSE Capability) escreveu: What is your Hardware Memory configuration for Main Storage and XSTORE for VM Larry Davis** *From:*The IBM z/VM Operating System [mailto:IBMVM@LISTSERV.UARK.EDU] *On Behalf Of *Carlos Bodra - Pessoal *Sent:* Wednesday, April 20, 2011 6:39 PM *To:* IBMVM@LISTSERV.UARK.EDU *Subject:* Re: z/VM RedHat Virtual Machine Memory abend Larry, We are running it z/VM 5.4 RSU 1003 (lastest), but problem is connected to Linux, because I changed MAINT user memory to 128GB and all runs fine. This machine has no CP´s just IFL´s processors. Carlos Bodra IBM Certified Specialist System z Sao Paulo - Brazil Em 20/04/2011 19:24, Davis, Larry (National VM/VSE Capability) escreveu: Are you Running VM on this or zLinux only on an LPAR? Larry Davis *From:*The IBM z/VM Operating System [mailto:IBMVM@LISTSERV.UARK.EDU] *On Behalf Of *Carlos Bodra - Pessoal *Sent:* Wednesday, April 20, 2011 6:19 PM *To:* IBMVM@LISTSERV.UARK.EDU mailto:IBMVM@LISTSERV.UARK.EDU *Subject:* z/VM RedHat Virtual Machine Memory abend Hi s390x Gurus I´m new to Linux on z and we are conducting a POC test of Linux Red Hat with Oracle running under z/VM 5.4 RSU 1003 and a z/10 BC model A00 (IFL Only). If linux virtual machine is defined with 64G in directory we got an abend during startup (see below). Doing some tests I found that I can define it until 59G in directory (60794036k from TOP linux command) and it will startup correctly. My questions are: 1 - Is this a limitation of linux s390x or Red Hat distribution? 2 - Is this a linux s390x bug or Red Hat distribution? 3 - How can I circunvent this so we can continue with POC (Proof Of Concept)? Thanks in advance for all hints and tips about. 00: Booting default (2.6.18-194.8.1.el5)... Linux version 2.6.18-194.8.1.el5 (mockbu...@s390-002.build.bos.redhat.com mailto:mockbu...@s390-002.build.bos.redhat.com) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-48)) #1 SMP Wed Jun 23 11:09:16 EDT 2010 We are running under VM (64 bit mode) Detected 3 CPU's Boot cpu address 0 Built 1 zonelists. Total pages: 16777216 Kernel command line: root=LABEL=/ BOOT_IMAGE=0 PID hash table entries: 4096 (order: 12, 32768 bytes) Dentry cache hash table entries: 8388608 (order: 14, 67108864 bytes) Inode-cache hash table entries: 4194304 (order: 13, 33554432 bytes) *Memory: 65951564k/67108864k available*(3414k kernel code, 0k reserved, 2256k dat a, 136k init) Write protected kernel read-only data: 0x356000 - 0x409fff Calibrating delay loop... 2981.88 BogoMIPS (lpj=14909440) Security Framework v1.0.0 initialized SELinux: Initializing. selinux_register_security: Registering secondary module capability Capability LSM initialized as secondary Mount-cache hash table entries: 256 cpu 0 phys_idx=0 vers=FF ident=01F2D6 machine=2098 unused=8000 cpu 1 phys_idx=1 vers=FF ident=01F2D6 machine=2098 unused=8000 cpu 2 phys_idx=2 vers=FF ident=01F2D6 machine=2098 unused=8000 Brought up 3 CPUs migration_cost=1000 checking if image is initramfs...it isn't (bad gzip magic numbers); looks like a n initrd Freeing initrd memory: 2759k freed list_del corruption. prev-next should be 8001bff0, but was 8001 c840 kernel BUG at lib/list_debug.c:65! illegal operation: 0001 Ý#1¨ CPU: 0 Not tainted 2.6.18-194.8.1.el5 #1 Process swapper (pid: 1, task: c611f788, ksp: c6123b10) Krnl PSW : 040400018000 002685d6 (list_del+0x9e/0xcc) Krnl GPRS: 0015 c6123cb8 0026 0400 0012a83e 0733 000a 0050e400 0001 8001bf90 8001bff0 8001bfce 00369878 002685d2 c6123d10 Krnl Code: d2 07 10 08 20 08 e3 10 d0 08 00 04 e3 10 20 08 00 24 e3 10 Call Trace: (Ý002685d2¨ list_del+0x9a/0xcc) Ý0017820a¨ free_pages_bulk+0x9e/0x34c Ý0017868a¨ free_hot_cold_page+0x1d2/0x1f4 Ý001009c0¨ free_initrd_mem+0x8c/0xc8 Ý005944ee¨ free_initrd+0xba/0xe4 Ý0058e324¨ init+0x1a0/0x32c Ý0010558e¨ kernel_thread_starter+0x6/0xc Ý00105588¨ kernel_thread_starter+0x0/0xc 0Kernel panic - not syncing: Fatal exception: panic_on_oops 01: HCPGSP2629I The virtual machine is placed in CP mode due to a SIGP stop from CPU 00. 02: HCPGSP2629I The virtual machine is placed in CP mode due to a SIGP stop from CPU 00. 00: HCPGIR450W CP entered; disabled wait PSW 00020001 8000 0010BDBE -- Carlos Bodra IBM Certified Specialist System z Sao Paulo - Brazil
Re: z/VM RedHat Virtual Machine Memory abend
Since you are overcommitting memory -- how much paging space do you have? Scott Rohling On Wed, Apr 20, 2011 at 5:32 PM, Carlos Bodra - Pessoal cbo...@terra.com.br wrote: q stor STORAGE = 48G CONFIGURED = 48G INC = 128M STANDBY = 0 RESERVED = 0 Ready; T=0.01/0.01 20:33:09 No xstore defined. Carlos Bodra IBM Certified Specialist System z Sao Paulo - Brazil Em 20/04/2011 20:04, Davis, Larry (National VM/VSE Capability) escreveu: What is your Hardware Memory configuration for Main Storage and XSTORE for VM Larry Davis** *From:* The IBM z/VM Operating System [mailto:IBMVM@LISTSERV.UARK.EDUIBMVM@LISTSERV.UARK.EDU] *On Behalf Of *Carlos Bodra - Pessoal *Sent:* Wednesday, April 20, 2011 6:39 PM *To:* IBMVM@LISTSERV.UARK.EDU *Subject:* Re: z/VM RedHat Virtual Machine Memory abend Larry, We are running it z/VM 5.4 RSU 1003 (lastest), but problem is connected to Linux, because I changed MAINT user memory to 128GB and all runs fine. This machine has no CP´s just IFL´s processors. Carlos Bodra IBM Certified Specialist System z Sao Paulo - Brazil Em 20/04/2011 19:24, Davis, Larry (National VM/VSE Capability) escreveu: Are you Running VM on this or zLinux only on an LPAR? Larry Davis *From:* The IBM z/VM Operating System [mailto:IBMVM@LISTSERV.UARK.EDUIBMVM@LISTSERV.UARK.EDU] *On Behalf Of *Carlos Bodra - Pessoal *Sent:* Wednesday, April 20, 2011 6:19 PM *To:* IBMVM@LISTSERV.UARK.EDU *Subject:* z/VM RedHat Virtual Machine Memory abend Hi s390x Gurus I´m new to Linux on z and we are conducting a POC test of Linux Red Hat with Oracle running under z/VM 5.4 RSU 1003 and a z/10 BC model A00 (IFL Only). If linux virtual machine is defined with 64G in directory we got an abend during startup (see below). Doing some tests I found that I can define it until 59G in directory (60794036k from TOP linux command) and it will startup correctly. My questions are: 1 - Is this a limitation of linux s390x or Red Hat distribution? 2 - Is this a linux s390x bug or Red Hat distribution? 3 - How can I circunvent this so we can continue with POC (Proof Of Concept)? Thanks in advance for all hints and tips about. 00: Booting default (2.6.18-194.8.1.el5)... Linux version 2.6.18-194.8.1.el5 (mockbu...@s390-002.build.bos.redhat.com) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-48)) #1 SMP Wed Jun 23 11:09:16 EDT 2010 We are running under VM (64 bit mode) Detected 3 CPU's Boot cpu address 0 Built 1 zonelists. Total pages: 16777216 Kernel command line: root=LABEL=/ BOOT_IMAGE=0 PID hash table entries: 4096 (order: 12, 32768 bytes) Dentry cache hash table entries: 8388608 (order: 14, 67108864 bytes) Inode-cache hash table entries: 4194304 (order: 13, 33554432 bytes) *Memory: 65951564k/67108864k available* (3414k kernel code, 0k reserved, 2256k dat a, 136k init) Write protected kernel read-only data: 0x356000 - 0x409fff Calibrating delay loop... 2981.88 BogoMIPS (lpj=14909440) Security Framework v1.0.0 initialized SELinux: Initializing. selinux_register_security: Registering secondary module capability Capability LSM initialized as secondary Mount-cache hash table entries: 256 cpu 0 phys_idx=0 vers=FF ident=01F2D6 machine=2098 unused=8000 cpu 1 phys_idx=1 vers=FF ident=01F2D6 machine=2098 unused=8000 cpu 2 phys_idx=2 vers=FF ident=01F2D6 machine=2098 unused=8000 Brought up 3 CPUs migration_cost=1000 checking if image is initramfs...it isn't (bad gzip magic numbers); looks like a n initrd Freeing initrd memory: 2759k freed list_del corruption. prev-next should be 8001bff0, but was 8001 c840 kernel BUG at lib/list_debug.c:65! illegal operation: 0001 Ý#1¨ CPU: 0 Not tainted 2.6.18-194.8.1.el5 #1 Process swapper (pid: 1, task: c611f788, ksp: c6123b10) Krnl PSW : 040400018000 002685d6 (list_del+0x9e/0xcc) Krnl GPRS: 0015 c6123cb8 0026 0400 0012a83e 0733 000a 0050e400 0001 8001bf90 8001bff0 8001bfce 00369878 002685d2 c6123d10 Krnl Code: d2 07 10 08 20 08 e3 10 d0 08 00 04 e3 10 20 08 00 24 e3 10 Call Trace: (Ý002685d2¨ list_del+0x9a/0xcc) Ý0017820a¨ free_pages_bulk+0x9e/0x34c Ý0017868a¨ free_hot_cold_page+0x1d2/0x1f4 Ý001009c0¨ free_initrd_mem+0x8c/0xc8 Ý005944ee¨ free_initrd+0xba/0xe4 Ý0058e324¨ init+0x1a0/0x32c Ý0010558e¨ kernel_thread_starter+0x6/0xc Ý00105588¨ kernel_thread_starter+0x0/0xc 0Kernel panic - not syncing: Fatal exception: panic_on_oops 01: HCPGSP2629I The virtual machine is placed in CP mode due to a SIGP stop from CPU 00. 02: HCPGSP2629I The virtual machine is placed in CP mode due to a SIGP stop from CPU 00. 00
Re: z/VM RedHat Virtual Machine Memory abend
Well that may be a problem VM always needs a little XSTORE unlike MVS and because you are trying to allocate a guest with more memory than you have Real Memory to support than VM will need someplace to Swap. How many Paging volumes do you have and what are their sizes. Also don't think that a WAS server on a zSeries box will need that much Storage. I would start him off with 32 GB maybe but give him some VDISK SWAP Space. Other s on this List can help with that and there is a Redpaper on setting up WebSphere on zLinux I attached it here, but if it doesn't make it search for Linux on IBM zSeries and S/390: z/VM Configuration for WebSphere Deployments Larry Davis From: The IBM z/VM Operating System [mailto:IBMVM@LISTSERV.UARK.EDU] On Behalf Of Carlos Bodra - Pessoal Sent: Wednesday, April 20, 2011 7:33 PM To: IBMVM@LISTSERV.UARK.EDU Subject: Re: z/VM RedHat Virtual Machine Memory abend q stor STORAGE = 48G CONFIGURED = 48G INC = 128M STANDBY = 0 RESERVED = 0 Ready; T=0.01/0.01 20:33:09 No xstore defined. Carlos Bodra IBM Certified Specialist System z Sao Paulo - Brazil Em 20/04/2011 20:04, Davis, Larry (National VM/VSE Capability) escreveu: What is your Hardware Memory configuration for Main Storage and XSTORE for VM Larry Davis From: The IBM z/VM Operating System [mailto:IBMVM@LISTSERV.UARK.EDU] On Behalf Of Carlos Bodra - Pessoal Sent: Wednesday, April 20, 2011 6:39 PM To: IBMVM@LISTSERV.UARK.EDUmailto:IBMVM@LISTSERV.UARK.EDU Subject: Re: z/VM RedHat Virtual Machine Memory abend Larry, We are running it z/VM 5.4 RSU 1003 (lastest), but problem is connected to Linux, because I changed MAINT user memory to 128GB and all runs fine. This machine has no CP´s just IFL´s processors. Carlos Bodra IBM Certified Specialist System z Sao Paulo - Brazil Em 20/04/2011 19:24, Davis, Larry (National VM/VSE Capability) escreveu: Are you Running VM on this or zLinux only on an LPAR? Larry Davis From: The IBM z/VM Operating System [mailto:IBMVM@LISTSERV.UARK.EDU] On Behalf Of Carlos Bodra - Pessoal Sent: Wednesday, April 20, 2011 6:19 PM To: IBMVM@LISTSERV.UARK.EDUmailto:IBMVM@LISTSERV.UARK.EDU Subject: z/VM RedHat Virtual Machine Memory abend Hi s390x Gurus I´m new to Linux on z and we are conducting a POC test of Linux Red Hat with Oracle running under z/VM 5.4 RSU 1003 and a z/10 BC model A00 (IFL Only). If linux virtual machine is defined with 64G in directory we got an abend during startup (see below). Doing some tests I found that I can define it until 59G in directory (60794036k from TOP linux command) and it will startup correctly. My questions are: 1 - Is this a limitation of linux s390x or Red Hat distribution? 2 - Is this a linux s390x bug or Red Hat distribution? 3 - How can I circunvent this so we can continue with POC (Proof Of Concept)? Thanks in advance for all hints and tips about. 00: Booting default (2.6.18-194.8.1.el5)... Linux version 2.6.18-194.8.1.el5 (mockbu...@s390-002.build.bos.redhat.commailto:mockbu...@s390-002.build.bos.redhat.com) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-48)) #1 SMP Wed Jun 23 11:09:16 EDT 2010 We are running under VM (64 bit mode) Detected 3 CPU's Boot cpu address 0 Built 1 zonelists. Total pages: 16777216 Kernel command line: root=LABEL=/ BOOT_IMAGE=0 PID hash table entries: 4096 (order: 12, 32768 bytes) Dentry cache hash table entries: 8388608 (order: 14, 67108864 bytes) Inode-cache hash table entries: 4194304 (order: 13, 33554432 bytes) Memory: 65951564k/67108864k available (3414k kernel code, 0k reserved, 2256k dat a, 136k init) Write protected kernel read-only data: 0x356000 - 0x409fff Calibrating delay loop... 2981.88 BogoMIPS (lpj=14909440) Security Framework v1.0.0 initialized SELinux: Initializing. selinux_register_security: Registering secondary module capability Capability LSM initialized as secondary Mount-cache hash table entries: 256 cpu 0 phys_idx=0 vers=FF ident=01F2D6 machine=2098 unused=8000 cpu 1 phys_idx=1 vers=FF ident=01F2D6 machine=2098 unused=8000 cpu 2 phys_idx=2 vers=FF ident=01F2D6 machine=2098 unused=8000 Brought up 3 CPUs migration_cost=1000 checking if image is initramfs...it isn't (bad gzip magic numbers); looks like a n initrd Freeing initrd memory: 2759k freed list_del corruption. prev-next should be 8001bff0, but was 8001 c840 kernel BUG at lib/list_debug.c:65! illegal operation: 0001 Ý#1¨ CPU: 0 Not tainted 2.6.18-194.8.1.el5 #1 Process swapper (pid: 1, task: c611f788, ksp: c6123b10) Krnl PSW : 040400018000 002685d6 (list_del+0x9e/0xcc) Krnl GPRS: 0015 c6123cb8 0026 0400 0012a83e 0733 000a 0050e400 0001 8001bf90 8001bff0 8001bfce 00369878 002685d2 c6123d10 Krnl Code: d2 07 10 08 20 08 e3 10 d0 08 00 04 e3 10 20 08 00 24 e3 10 Call Trace
Re: z/VM RedHat Virtual Machine Memory abend
I would also wonder where you came up with that 64G virtual memory requirement. The rules of thumb that apply to intel environments are counterproductive in the Z environment. I hope our Velocity friends will expound at length about that. From: The IBM z/VM Operating System [mailto:IBMVM@LISTSERV.UARK.EDU] On Behalf Of Davis, Larry (National VM/VSE Capability) Sent: Wednesday, April 20, 2011 7:55 PM To: IBMVM@LISTSERV.UARK.EDU Subject: Re: z/VM RedHat Virtual Machine Memory abend Well that may be a problem VM always needs a little XSTORE unlike MVS and because you are trying to allocate a guest with more memory than you have Real Memory to support than VM will need someplace to Swap. How many Paging volumes do you have and what are their sizes. Also don't think that a WAS server on a zSeries box will need that much Storage. I would start him off with 32 GB maybe but give him some VDISK SWAP Space. Other s on this List can help with that and there is a Redpaper on setting up WebSphere on zLinux I attached it here, but if it doesn't make it search for Linux on IBM zSeries and S/390: z/VM Configuration for WebSphere Deployments Larry Davis From: The IBM z/VM Operating System [mailto:IBMVM@LISTSERV.UARK.EDU] On Behalf Of Carlos Bodra - Pessoal Sent: Wednesday, April 20, 2011 7:33 PM To: IBMVM@LISTSERV.UARK.EDU Subject: Re: z/VM RedHat Virtual Machine Memory abend q stor STORAGE = 48G CONFIGURED = 48G INC = 128M STANDBY = 0 RESERVED = 0 Ready; T=0.01/0.01 20:33:09 No xstore defined. Carlos Bodra IBM Certified Specialist System z Sao Paulo - Brazil Em 20/04/2011 20:04, Davis, Larry (National VM/VSE Capability) escreveu: What is your Hardware Memory configuration for Main Storage and XSTORE for VM Larry Davis From: The IBM z/VM Operating System [mailto:IBMVM@LISTSERV.UARK.EDU] On Behalf Of Carlos Bodra - Pessoal Sent: Wednesday, April 20, 2011 6:39 PM To: IBMVM@LISTSERV.UARK.EDU Subject: Re: z/VM RedHat Virtual Machine Memory abend Larry, We are running it z/VM 5.4 RSU 1003 (lastest), but problem is connected to Linux, because I changed MAINT user memory to 128GB and all runs fine. This machine has no CP´s just IFL´s processors. Carlos Bodra IBM Certified Specialist System z Sao Paulo - Brazil Em 20/04/2011 19:24, Davis, Larry (National VM/VSE Capability) escreveu: Are you Running VM on this or zLinux only on an LPAR? Larry Davis From: The IBM z/VM Operating System [mailto:IBMVM@LISTSERV.UARK.EDU] On Behalf Of Carlos Bodra - Pessoal Sent: Wednesday, April 20, 2011 6:19 PM To: IBMVM@LISTSERV.UARK.EDU Subject: z/VM RedHat Virtual Machine Memory abend Hi s390x Gurus I´m new to Linux on z and we are conducting a POC test of Linux Red Hat with Oracle running under z/VM 5.4 RSU 1003 and a z/10 BC model A00 (IFL Only). If linux virtual machine is defined with 64G in directory we got an abend during startup (see below). Doing some tests I found that I can define it until 59G in directory (60794036k from TOP linux command) and it will startup correctly. My questions are: 1 - Is this a limitation of linux s390x or Red Hat distribution? 2 - Is this a linux s390x bug or Red Hat distribution? 3 - How can I circunvent this so we can continue with POC (Proof Of Concept)? Thanks in advance for all hints and tips about. 00: Booting default (2.6.18-194.8.1.el5)... Linux version 2.6.18-194.8.1.el5 (mockbu...@s390-002.build.bos.redhat.com) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-48)) #1 SMP Wed Jun 23 11:09:16 EDT 2010 We are running under VM (64 bit mode) Detected 3 CPU's Boot cpu address 0 Built 1 zonelists. Total pages: 16777216 Kernel command line: root=LABEL=/ BOOT_IMAGE=0 PID hash table entries: 4096 (order: 12, 32768 bytes) Dentry cache hash table entries: 8388608 (order: 14, 67108864 bytes) Inode-cache hash table entries: 4194304 (order: 13, 33554432 bytes) Memory: 65951564k/67108864k available (3414k kernel code, 0k reserved, 2256k dat a, 136k init) Write protected kernel read-only data: 0x356000 - 0x409fff Calibrating delay loop... 2981.88 BogoMIPS (lpj=14909440) Security Framework v1.0.0 initialized SELinux: Initializing. selinux_register_security: Registering secondary module capability
Re: z/VM RedHat Virtual Machine Memory abend
I didn't say anything at first because clearly there IS a problem. Linux is crashing when presented with 64G of storage, but not when presented with 59G, per what Carlos said at the start. This is not likely a VM or hardware problem. True, the req is probably ROT and not based on actual application needs but on platform assumptions. If 59G IPLs, then run the POC with that and see what happens. In production you will almost certainly want to REDUCE the memory given to each v-machine. 59G is typically too much, sometimes *way* too much. Virtualized Linux has been around for more than ten years, but many consultants and app vendors still don't understand. -- Rick, Velocity friend; On Apr 20, 2011 8:04 PM, Quay, Jonathan (IHG) jonathan.q...@ihg.com wrote:
Re: z/VM RedHat Virtual Machine Memory abend
Check the amount of paging space you have defined for the VM system. You can define a virtual machine far bigger than you can actually use if you have insufficient paging space. The guest will work for a while, until it tries to access a page that can't be supported with backing store, and you get weird memory problems like that. Also, seriously ask why the virtual machine needs to be so large. If it's because that's the way it is on the distributed platform, that's going to do more harm than good - I/O avoidance is not as important on this platform as it is elsewhere (the biggest reason for huge memory sizes). Start with 1 or 2G, reduce the size of SGA, and make sure you use VDISK for swap. Also make sure you have some XSTORE defined if you're going to have virtual machines that big - you're going to need it for setting up a paging hierarchy that can sustain moving
Re: z/VM RedHat Virtual Machine Memory abend
Check the amount of paging space you have defined for the VM system. Is this a brand new VM install right out of the box? If so, then this is most likely to be the problem. 48G of real, plus the default paging areas in a brand new VM install add up to just about 52G or so, depending on whether you used mod 3 or mod 9s. It works for MAINT because CMS is extremely memory efficient; it doesn't require that every page in the virtual machine definition be actually present when it IPLs, and it touches only what it actually needs. Linux touches every page at some point, so you need enough paging space to support the total size of all the virtual machines you define, plus a little bit of insurance. Add at least 3-4 mod 9 or mod 27s as paging space and see if the problem goes away.
Re: z/VM RedHat Virtual Machine Memory abend
Thanks a lot for hints. Barton, and others, I will redifine storage areas to get xstore. I can add more 2 or 3 3390-3 page volumes. Today there is only one volume for page and one for spool. Please keep in mind, that this is a no production z10 installed in our data center just for POC tests, so I can change definitions very very fast. I will do changes next monday and return here with results. Thanks again and Happy Easter for all!!! Carlos Bodra On Qua 20/04/11 22:52 , David Boyes dbo...@sinenomine.net sent: Check the amount of paging space you have defined for the VM system. Is this a brand new VM install right out of the box? If so, then this is most likely to be the problem. 48G of real, plus the default paging areas in a brand new VM install add up to just about 52G or so, depending on whether you used mod 3 or mod 9s. It works for MAINT because CMS is extremely memory efficient; it doesn’t require that every page in the virtual machine definition be actually present when it IPLs, and it touches only what it actually needs. Linux touches every page at some point, so you need enough paging space to support the total size of all the virtual machines you define, plus a little bit of insurance. Add at least 3-4 mod 9 or mod 27s as paging space and see if the problem goes away.
