Re: [RFC PATCH v5 1/9] fadump: Add documentation for firmware-assisted dump.
On 11/25/2011 04:04 AM, Paul Mackerras wrote: >> + /sys/kernel/debug/powerpc/fadump_region >> + >> +This file shows the reserved memory regions if fadump is >> +enabled otherwise this file is empty. The output format >> +is: >> +: [-] bytes, Dumped: >> + >> +e.g. >> +Contents when fadump is registered during first kernel >> + >> +# cat /sys/kernel/debug/powerpc/fadump_region >> +CPU : [0x006ffb-0x006fff001f] 0x40020 bytes, Dumped: 0x0 >> +HPTE: [0x006fff0020-0x006fff101f] 0x1000 bytes, Dumped: 0x0 >> +DUMP: [0x006fff1020-0x007fff101f] 0x1000 bytes, Dumped: 0x0 > > How come the HPTE region is only 0x1000 (4k) bytes? The hashed page > table (HPT) will be much bigger than this. Is this our way of telling > the hypervisor that we don't care about the HPT? If so, is it > possible to make this region 0 bytes instead of 0x1000? > The firmware assisted dump registration fails with "Hardware error (-1)" when called with HPTE dump section with size 0. According to PAPR the size returned by ibm,configure-kernel-dump-sizes node for HPTE dump section is a minimum size for HPTE dump section. Though PAPR dose not mention this, but after sevaral trial and error, it looks like the HPTE dump section is one of the mandatory argument that needs to be passed (with non-zero size) while making ibm,configure-kernel-dump rtas call. Thanks, -Mahesh. ___ Linuxppc-dev mailing list Linuxppc-dev@lists.ozlabs.org https://lists.ozlabs.org/listinfo/linuxppc-dev
Re: [RFC PATCH v5 1/9] fadump: Add documentation for firmware-assisted dump.
On 11/25/2011 04:04 AM, Paul Mackerras wrote: > On Tue, Nov 15, 2011 at 08:43:34PM +0530, Mahesh J Salgaonkar wrote: >> From: Mahesh Salgaonkar >> >> Documentation for firmware-assisted dump. This document is based on the >> original documentation written for phyp assisted dump by Linas Vepstas >> and Manish Ahuja, with few changes to reflect the current implementation. >> >> Change in v3: >> - Modified the documentation to reflect introdunction of fadump_registered >> sysfs file and few minor changes. >> >> Change in v2: >> - Modified the documentation to reflect the change of fadump_region >> file under debugfs filesystem. > > In general we don't want the changes between successive versions in > the patch description; this information should go below the "---" > line. The patch description should describe how the patch is now and > give any information that will be useful to someone looking at the > resulting git commit later on, but it doesn't need to tell us about > previous versions of the patch that will never appear in the git > history. Sure will do that. > >> +-- Once the dump is copied out, the memory that held the dump >> + is immediately available to the running kernel. A further >> + reboot isn't required. > > I have a general worry about the system making allocations that are > intended to be node-local while it is running with restricted memory > (i.e. after the crash and reboot and before the dump has been written > out and the dump memory freed). Those allocations will probably all > come from one node and thus won't necessarily be on the desired node. > So, for very large systems with significant NUMA characteristics, it > may be desirable (though not required) to reboot after taking the > dump. I have been working on trying to integrate FADUMP with the kdump infrastructure on distros, which will modify the existing kernel initrd to capture the vmcore and release the memory at the very early stage before the switch_root. However, by default FADUMP will also reboot after capturing vmcore unless user specifies 'noreboot' option through kdump configuration file. > > What happens about the NUMA information in the kernel -- all the > memory sections, etc.? Do they get set up as normal even though the > second kernel is booting with only a small amount of memory initially? > In FADUMP case, the booting of second kernel after crash is equivalent to normal kernel bootup and it boots with the knowledge of entire system RAM with NUMA information. The memblock structure does contain map for entire system RAM. We just reserve the memory above the bootmem at the very early stage in the second kernel, so that it remains untouched. Thanks, -Mahesh. ___ Linuxppc-dev mailing list Linuxppc-dev@lists.ozlabs.org https://lists.ozlabs.org/listinfo/linuxppc-dev
Re: [RFC PATCH v5 1/9] fadump: Add documentation for firmware-assisted dump.
On Tue, Nov 15, 2011 at 08:43:34PM +0530, Mahesh J Salgaonkar wrote: > From: Mahesh Salgaonkar > > Documentation for firmware-assisted dump. This document is based on the > original documentation written for phyp assisted dump by Linas Vepstas > and Manish Ahuja, with few changes to reflect the current implementation. > > Change in v3: > - Modified the documentation to reflect introdunction of fadump_registered > sysfs file and few minor changes. > > Change in v2: > - Modified the documentation to reflect the change of fadump_region > file under debugfs filesystem. In general we don't want the changes between successive versions in the patch description; this information should go below the "---" line. The patch description should describe how the patch is now and give any information that will be useful to someone looking at the resulting git commit later on, but it doesn't need to tell us about previous versions of the patch that will never appear in the git history. > +-- Once the dump is copied out, the memory that held the dump > + is immediately available to the running kernel. A further > + reboot isn't required. I have a general worry about the system making allocations that are intended to be node-local while it is running with restricted memory (i.e. after the crash and reboot and before the dump has been written out and the dump memory freed). Those allocations will probably all come from one node and thus won't necessarily be on the desired node. So, for very large systems with significant NUMA characteristics, it may be desirable (though not required) to reboot after taking the dump. What happens about the NUMA information in the kernel -- all the memory sections, etc.? Do they get set up as normal even though the second kernel is booting with only a small amount of memory initially? > + /sys/kernel/debug/powerpc/fadump_region > + > +This file shows the reserved memory regions if fadump is > +enabled otherwise this file is empty. The output format > +is: > +: [-] bytes, Dumped: > + > +e.g. > +Contents when fadump is registered during first kernel > + > +# cat /sys/kernel/debug/powerpc/fadump_region > +CPU : [0x006ffb-0x006fff001f] 0x40020 bytes, Dumped: 0x0 > +HPTE: [0x006fff0020-0x006fff101f] 0x1000 bytes, Dumped: 0x0 > +DUMP: [0x006fff1020-0x007fff101f] 0x1000 bytes, Dumped: 0x0 How come the HPTE region is only 0x1000 (4k) bytes? The hashed page table (HPT) will be much bigger than this. Is this our way of telling the hypervisor that we don't care about the HPT? If so, is it possible to make this region 0 bytes instead of 0x1000? Paul. ___ Linuxppc-dev mailing list Linuxppc-dev@lists.ozlabs.org https://lists.ozlabs.org/listinfo/linuxppc-dev
[RFC PATCH v5 1/9] fadump: Add documentation for firmware-assisted dump.
From: Mahesh Salgaonkar Documentation for firmware-assisted dump. This document is based on the original documentation written for phyp assisted dump by Linas Vepstas and Manish Ahuja, with few changes to reflect the current implementation. Change in v3: - Modified the documentation to reflect introdunction of fadump_registered sysfs file and few minor changes. Change in v2: - Modified the documentation to reflect the change of fadump_region file under debugfs filesystem. Signed-off-by: Mahesh Salgaonkar --- Documentation/powerpc/firmware-assisted-dump.txt | 243 ++ 1 files changed, 243 insertions(+), 0 deletions(-) create mode 100644 Documentation/powerpc/firmware-assisted-dump.txt diff --git a/Documentation/powerpc/firmware-assisted-dump.txt b/Documentation/powerpc/firmware-assisted-dump.txt new file mode 100644 index 000..3248b5d --- /dev/null +++ b/Documentation/powerpc/firmware-assisted-dump.txt @@ -0,0 +1,243 @@ + + Firmware-Assisted Dump + + July 2011 + +The goal of firmware-assisted dump is to enable the dump of +a crashed system, and to do so from a fully-reset system, and +to minimize the total elapsed time until the system is back +in production use. + +Comparing with kdump or other strategies, firmware-assisted +dump offers several strong, practical advantages: + +-- Unlike kdump, the system has been reset, and loaded + with a fresh copy of the kernel. In particular, + PCI and I/O devices have been reinitialized and are + in a clean, consistent state. +-- Once the dump is copied out, the memory that held the dump + is immediately available to the running kernel. A further + reboot isn't required. + +The above can only be accomplished by coordination with, +and assistance from the Power firmware. The procedure is +as follows: + +-- The first kernel registers the sections of memory with the + Power firmware for dump preservation during OS initialization. + These registered sections of memory are reserved by the first + kernel during early boot. + +-- When a system crashes, the Power firmware will save + the low memory (boot memory of size larger of 5% of system RAM + or 256MB) of RAM to the previous registered region. It will + also save system registers, and hardware PTE's. + + NOTE: The term 'boot memory' means size of the low memory chunk + that is required for a kernel to boot successfully when + booted with restricted memory. By default, the boot memory + size will be the larger of 5% of system RAM or 256MB. + Alternatively, user can also specify boot memory size + through boot parameter 'fadump_reserve_mem=' which will + override the default calculated size. + +-- After the low memory (boot memory) area has been saved, the + firmware will reset PCI and other hardware state. It will + *not* clear the RAM. It will then launch the bootloader, as + normal. + +-- The freshly booted kernel will notice that there is a new + node (ibm,dump-kernel) in the device tree, indicating that + there is crash data available from a previous boot. During + the early boot OS will reserve rest of the memory above + boot memory size effectively booting with restricted memory + size. This will make sure that the second kernel will not + touch any of the dump memory area. + +-- User-space tools will read /proc/vmcore to obtain the contents + of memory, which holds the previous crashed kernel dump in ELF + format. The userspace tools may copy this info to disk, or + network, nas, san, iscsi, etc. as desired. + +-- Once the userspace tool is done saving dump, it will echo + '1' to /sys/kernel/fadump_release_mem to release the reserved + memory back to general use, except the memory required for + next firmware-assisted dump registration. + + e.g. + # echo 1 > /sys/kernel/fadump_release_mem + +Please note that the firmware-assisted dump feature +is only available on Power6 and above systems with recent +firmware versions. + +Implementation details: +-- + +During boot, a check is made to see if firmware supports +this feature on that particular machine. If it does, then +we check to see if an active dump is waiting for us. If yes +then everything but boot memory size of RAM is reserved during +early boot (See Fig. 2). This area is released once we finish +collecting the dump from user land scripts (e.g. kdump scripts) +that are run. If there is dump data, then the +/sys/kernel/fadump_release_mem file is created, and the reserved +memory is held. + +If there is no waiting dump data, then only the memory required +to hold CPU state, HPTE region, boot memory dump and elfcore +header, is reserved at the top of memory (see Fig. 1). This area +is *not* released: this region will be kept permanently reserved, +so that it can act as a receptacle for a copy of the boot memory +con
