Re: [PATCH v8 0/7] crash: Kernel handling of CPU and memory hot un/plug
On 5/31/22 08:18, David Hildenbrand wrote:
On 26.05.22 15:39, Sourabh Jain wrote:
Hello Eric,
On 26/05/22 18:46, Eric DeVolder wrote:
On 5/25/22 10:13, Sourabh Jain wrote:
Hello Eric,
On 06/05/22 00:15, Eric DeVolder wrote:
When the kdump service is loaded, if a CPU or memory is hot
un/plugged, the crash elfcorehdr (for x86), which describes the CPUs
and memory in the system, must also be updated, else the resulting
vmcore is inaccurate (eg. missing either CPU context or memory
regions).
The current solution utilizes udev to initiate an unload-then-reload
of the kdump image (e. kernel, initrd, boot_params, puratory and
elfcorehdr) by the userspace kexec utility. In previous posts I have
outlined the significant performance problems related to offloading
this activity to userspace.
This patchset introduces a generic crash hot un/plug handler that
registers with the CPU and memory notifiers. Upon CPU or memory
changes, this generic handler is invoked and performs important
housekeeping, for example obtaining the appropriate lock, and then
invokes an architecture specific handler to do the appropriate
updates.
In the case of x86_64, the arch specific handler generates a new
elfcorehdr, and overwrites the old one in memory. No involvement
with userspace needed.
To realize the benefits/test this patchset, one must make a couple
of minor changes to userspace:
- Disable the udev rule for updating kdump on hot un/plug changes.
Add the following as the first two lines to the udev rule file
/usr/lib/udev/rules.d/98-kexec.rules:
If we can have a sysfs attribute to advertise this feature then
userspace
utilities (kexec tool/udev rules) can take action accordingly. In
short, it will
help us maintain backward compatibility.
kexec tool can use the new sysfs attribute and allocate additional
buffer space
for elfcorehdr accordingly. Similarly, the checksum-related changes
can come
under this check.
Udev rule can use this sysfs file to decide kdump service reload is
required or not.
Great idea. I've been working on the corresponding udev and
kexec-tools changes and your input/idea here is quite timely.
I have boolean "crash_hotplug" as a core_param(), so it will show up as:
# cat /sys/module/kernel/parameters/crash_hotplug
N
How about using 0-1 instead Y/N?
0 = crash hotplug not supported
1 = crash hotplug supported
Also how about keeping sysfs here instead?
/sys/kernel/kexec_crash_hotplug
It's not only about hotplug, though. And actually we care about
onlining/offlining. Hmm, I wonder if there is a better name for this
automatic handling of cpu and memory devices.
In the upcoming v9, there is no /sys/kernel/crash/kexec_crash_hotplug; I have sysfs attributes for
memory blocks and CPUs named 'crash_hotplug' that can be utilized directly in udev rule as
ATTR{crash_hotplug} to determine if the kernel is handling this for crash kernel update purposes.
Here's the current commit message for that change:
crash: memory and CPU hotplug sysfs attributes
This introduces the crash_hotplug attribute for memory and CPUs
for use by userspace. This change directly facilitates the udev
rule for managing userspace re-loading of the crash kernel.
For memory, this changeset introduces the crash_hotplug attribute
to the /sys/devices/system/memory directory. For example:
# udevadm info --attribute-walk /sys/devices/system/memory/memory81
looking at device '/devices/system/memory/memory81':
KERNEL=="memory81"
SUBSYSTEM=="memory"
DRIVER==""
ATTR{online}=="1"
ATTR{phys_device}=="0"
ATTR{phys_index}=="0051"
ATTR{removable}=="1"
ATTR{state}=="online"
ATTR{valid_zones}=="Movable"
looking at parent device '/devices/system/memory':
KERNELS=="memory"
SUBSYSTEMS==""
DRIVERS==""
ATTRS{auto_online_blocks}=="offline"
ATTRS{block_size_bytes}=="800"
ATTRS{crash_hotplug}=="1"
For CPUs, this changeset introduces the crash_hotplug attribute
to the /sys/devices/system/cpu directory. For example:
# udevadm info --attribute-walk /sys/devices/system/cpu/cpu0
looking at device '/devices/system/cpu/cpu0':
KERNEL=="cpu0"
SUBSYSTEM=="cpu"
DRIVER=="processor"
ATTR{crash_notes}=="277c38600"
ATTR{crash_notes_size}=="368"
ATTR{online}=="1"
looking at parent device '/devices/system/cpu':
KERNELS=="cpu"
SUBSYSTEMS==""
DRIVERS==""
ATTRS{crash_hotplug}=="1"
ATTRS{isolated}==""
ATTRS{kernel_max}=="8191"
ATTRS{nohz_full}==" (null)"
ATTRS{offline}=="4-7"
ATTRS{online}=="0-3"
ATTRS{possible}=="0-7"
ATTRS{present}=="0-3"
With these changes in place, and by using the same attribute
crash_hotplug name, it is possible to efficiently instruct the
udev rule to skip crash kernel reloading.
For example, the following is the proposed udev rule change for RHEL
system 98-kexec.rules (as the first two lines of the rule file):
# The kernel handles updates to crash elfcorehdr
ATTRS{cras
Re: [PATCH v8 0/7] crash: Kernel handling of CPU and memory hot un/plug
On 26.05.22 15:39, Sourabh Jain wrote: > Hello Eric, > > On 26/05/22 18:46, Eric DeVolder wrote: >> >> >> On 5/25/22 10:13, Sourabh Jain wrote: >>> Hello Eric, >>> >>> On 06/05/22 00:15, Eric DeVolder wrote: When the kdump service is loaded, if a CPU or memory is hot un/plugged, the crash elfcorehdr (for x86), which describes the CPUs and memory in the system, must also be updated, else the resulting vmcore is inaccurate (eg. missing either CPU context or memory regions). The current solution utilizes udev to initiate an unload-then-reload of the kdump image (e. kernel, initrd, boot_params, puratory and elfcorehdr) by the userspace kexec utility. In previous posts I have outlined the significant performance problems related to offloading this activity to userspace. This patchset introduces a generic crash hot un/plug handler that registers with the CPU and memory notifiers. Upon CPU or memory changes, this generic handler is invoked and performs important housekeeping, for example obtaining the appropriate lock, and then invokes an architecture specific handler to do the appropriate updates. In the case of x86_64, the arch specific handler generates a new elfcorehdr, and overwrites the old one in memory. No involvement with userspace needed. To realize the benefits/test this patchset, one must make a couple of minor changes to userspace: - Disable the udev rule for updating kdump on hot un/plug changes. Add the following as the first two lines to the udev rule file /usr/lib/udev/rules.d/98-kexec.rules: >>> >>> If we can have a sysfs attribute to advertise this feature then >>> userspace >>> utilities (kexec tool/udev rules) can take action accordingly. In >>> short, it will >>> help us maintain backward compatibility. >>> >>> kexec tool can use the new sysfs attribute and allocate additional >>> buffer space >>> for elfcorehdr accordingly. Similarly, the checksum-related changes >>> can come >>> under this check. >>> >>> Udev rule can use this sysfs file to decide kdump service reload is >>> required or not. >> >> Great idea. I've been working on the corresponding udev and >> kexec-tools changes and your input/idea here is quite timely. >> >> I have boolean "crash_hotplug" as a core_param(), so it will show up as: >> >> # cat /sys/module/kernel/parameters/crash_hotplug >> N > > How about using 0-1 instead Y/N? > 0 = crash hotplug not supported > 1 = crash hotplug supported > > Also how about keeping sysfs here instead? > /sys/kernel/kexec_crash_hotplug It's not only about hotplug, though. And actually we care about onlining/offlining. Hmm, I wonder if there is a better name for this automatic handling of cpu and memory devices. -- Thanks, David / dhildenb ___ kexec mailing list kexec@lists.infradead.org http://lists.infradead.org/mailman/listinfo/kexec
Re: [PATCH v8 0/7] crash: Kernel handling of CPU and memory hot un/plug
Hello Eric, On 26/05/22 18:46, Eric DeVolder wrote: On 5/25/22 10:13, Sourabh Jain wrote: Hello Eric, On 06/05/22 00:15, Eric DeVolder wrote: When the kdump service is loaded, if a CPU or memory is hot un/plugged, the crash elfcorehdr (for x86), which describes the CPUs and memory in the system, must also be updated, else the resulting vmcore is inaccurate (eg. missing either CPU context or memory regions). The current solution utilizes udev to initiate an unload-then-reload of the kdump image (e. kernel, initrd, boot_params, puratory and elfcorehdr) by the userspace kexec utility. In previous posts I have outlined the significant performance problems related to offloading this activity to userspace. This patchset introduces a generic crash hot un/plug handler that registers with the CPU and memory notifiers. Upon CPU or memory changes, this generic handler is invoked and performs important housekeeping, for example obtaining the appropriate lock, and then invokes an architecture specific handler to do the appropriate updates. In the case of x86_64, the arch specific handler generates a new elfcorehdr, and overwrites the old one in memory. No involvement with userspace needed. To realize the benefits/test this patchset, one must make a couple of minor changes to userspace: - Disable the udev rule for updating kdump on hot un/plug changes. Add the following as the first two lines to the udev rule file /usr/lib/udev/rules.d/98-kexec.rules: If we can have a sysfs attribute to advertise this feature then userspace utilities (kexec tool/udev rules) can take action accordingly. In short, it will help us maintain backward compatibility. kexec tool can use the new sysfs attribute and allocate additional buffer space for elfcorehdr accordingly. Similarly, the checksum-related changes can come under this check. Udev rule can use this sysfs file to decide kdump service reload is required or not. Great idea. I've been working on the corresponding udev and kexec-tools changes and your input/idea here is quite timely. I have boolean "crash_hotplug" as a core_param(), so it will show up as: # cat /sys/module/kernel/parameters/crash_hotplug N How about using 0-1 instead Y/N? 0 = crash hotplug not supported 1 = crash hotplug supported Also how about keeping sysfs here instead? /sys/kernel/kexec_crash_hotplug Thanks, Souabh Jain ___ kexec mailing list kexec@lists.infradead.org http://lists.infradead.org/mailman/listinfo/kexec
Re: [PATCH v8 0/7] crash: Kernel handling of CPU and memory hot un/plug
On 5/26/22 08:39, Sourabh Jain wrote: Hello Eric, On 26/05/22 18:46, Eric DeVolder wrote: On 5/25/22 10:13, Sourabh Jain wrote: Hello Eric, On 06/05/22 00:15, Eric DeVolder wrote: When the kdump service is loaded, if a CPU or memory is hot un/plugged, the crash elfcorehdr (for x86), which describes the CPUs and memory in the system, must also be updated, else the resulting vmcore is inaccurate (eg. missing either CPU context or memory regions). The current solution utilizes udev to initiate an unload-then-reload of the kdump image (e. kernel, initrd, boot_params, puratory and elfcorehdr) by the userspace kexec utility. In previous posts I have outlined the significant performance problems related to offloading this activity to userspace. This patchset introduces a generic crash hot un/plug handler that registers with the CPU and memory notifiers. Upon CPU or memory changes, this generic handler is invoked and performs important housekeeping, for example obtaining the appropriate lock, and then invokes an architecture specific handler to do the appropriate updates. In the case of x86_64, the arch specific handler generates a new elfcorehdr, and overwrites the old one in memory. No involvement with userspace needed. To realize the benefits/test this patchset, one must make a couple of minor changes to userspace: - Disable the udev rule for updating kdump on hot un/plug changes. Add the following as the first two lines to the udev rule file /usr/lib/udev/rules.d/98-kexec.rules: If we can have a sysfs attribute to advertise this feature then userspace utilities (kexec tool/udev rules) can take action accordingly. In short, it will help us maintain backward compatibility. kexec tool can use the new sysfs attribute and allocate additional buffer space for elfcorehdr accordingly. Similarly, the checksum-related changes can come under this check. Udev rule can use this sysfs file to decide kdump service reload is required or not. Great idea. I've been working on the corresponding udev and kexec-tools changes and your input/idea here is quite timely. I have boolean "crash_hotplug" as a core_param(), so it will show up as: # cat /sys/module/kernel/parameters/crash_hotplug N How about using 0-1 instead Y/N? 0 = crash hotplug not supported 1 = crash hotplug supported Also how about keeping sysfs here instead? /sys/kernel/kexec_crash_hotplug Yes, that makes more sense. Thanks! eric Thanks, Souabh Jain ___ kexec mailing list kexec@lists.infradead.org http://lists.infradead.org/mailman/listinfo/kexec
Re: [PATCH v8 0/7] crash: Kernel handling of CPU and memory hot un/plug
On 5/25/22 10:13, Sourabh Jain wrote: Hello Eric, On 06/05/22 00:15, Eric DeVolder wrote: When the kdump service is loaded, if a CPU or memory is hot un/plugged, the crash elfcorehdr (for x86), which describes the CPUs and memory in the system, must also be updated, else the resulting vmcore is inaccurate (eg. missing either CPU context or memory regions). The current solution utilizes udev to initiate an unload-then-reload of the kdump image (e. kernel, initrd, boot_params, puratory and elfcorehdr) by the userspace kexec utility. In previous posts I have outlined the significant performance problems related to offloading this activity to userspace. This patchset introduces a generic crash hot un/plug handler that registers with the CPU and memory notifiers. Upon CPU or memory changes, this generic handler is invoked and performs important housekeeping, for example obtaining the appropriate lock, and then invokes an architecture specific handler to do the appropriate updates. In the case of x86_64, the arch specific handler generates a new elfcorehdr, and overwrites the old one in memory. No involvement with userspace needed. To realize the benefits/test this patchset, one must make a couple of minor changes to userspace: - Disable the udev rule for updating kdump on hot un/plug changes. Add the following as the first two lines to the udev rule file /usr/lib/udev/rules.d/98-kexec.rules: If we can have a sysfs attribute to advertise this feature then userspace utilities (kexec tool/udev rules) can take action accordingly. In short, it will help us maintain backward compatibility. kexec tool can use the new sysfs attribute and allocate additional buffer space for elfcorehdr accordingly. Similarly, the checksum-related changes can come under this check. Udev rule can use this sysfs file to decide kdump service reload is required or not. Great idea. I've been working on the corresponding udev and kexec-tools changes and your input/idea here is quite timely. I have boolean "crash_hotplug" as a core_param(), so it will show up as: # cat /sys/module/kernel/parameters/crash_hotplug N This will provide userspace the indication it needs. Thanks, Sourabh Jain ___ kexec mailing list kexec@lists.infradead.org http://lists.infradead.org/mailman/listinfo/kexec
Re: [PATCH v8 0/7] crash: Kernel handling of CPU and memory hot un/plug
Hello Eric, On 06/05/22 00:15, Eric DeVolder wrote: When the kdump service is loaded, if a CPU or memory is hot un/plugged, the crash elfcorehdr (for x86), which describes the CPUs and memory in the system, must also be updated, else the resulting vmcore is inaccurate (eg. missing either CPU context or memory regions). The current solution utilizes udev to initiate an unload-then-reload of the kdump image (e. kernel, initrd, boot_params, puratory and elfcorehdr) by the userspace kexec utility. In previous posts I have outlined the significant performance problems related to offloading this activity to userspace. This patchset introduces a generic crash hot un/plug handler that registers with the CPU and memory notifiers. Upon CPU or memory changes, this generic handler is invoked and performs important housekeeping, for example obtaining the appropriate lock, and then invokes an architecture specific handler to do the appropriate updates. In the case of x86_64, the arch specific handler generates a new elfcorehdr, and overwrites the old one in memory. No involvement with userspace needed. To realize the benefits/test this patchset, one must make a couple of minor changes to userspace: - Disable the udev rule for updating kdump on hot un/plug changes. Add the following as the first two lines to the udev rule file /usr/lib/udev/rules.d/98-kexec.rules: If we can have a sysfs attribute to advertise this feature then userspace utilities (kexec tool/udev rules) can take action accordingly. In short, it will help us maintain backward compatibility. kexec tool can use the new sysfs attribute and allocate additional buffer space for elfcorehdr accordingly. Similarly, the checksum-related changes can come under this check. Udev rule can use this sysfs file to decide kdump service reload is required or not. Thanks, Sourabh Jain ___ kexec mailing list kexec@lists.infradead.org http://lists.infradead.org/mailman/listinfo/kexec
[PATCH v8 0/7] crash: Kernel handling of CPU and memory hot un/plug
When the kdump service is loaded, if a CPU or memory is hot
un/plugged, the crash elfcorehdr (for x86), which describes the CPUs
and memory in the system, must also be updated, else the resulting
vmcore is inaccurate (eg. missing either CPU context or memory
regions).
The current solution utilizes udev to initiate an unload-then-reload
of the kdump image (e. kernel, initrd, boot_params, puratory and
elfcorehdr) by the userspace kexec utility. In previous posts I have
outlined the significant performance problems related to offloading
this activity to userspace.
This patchset introduces a generic crash hot un/plug handler that
registers with the CPU and memory notifiers. Upon CPU or memory
changes, this generic handler is invoked and performs important
housekeeping, for example obtaining the appropriate lock, and then
invokes an architecture specific handler to do the appropriate
updates.
In the case of x86_64, the arch specific handler generates a new
elfcorehdr, and overwrites the old one in memory. No involvement
with userspace needed.
To realize the benefits/test this patchset, one must make a couple
of minor changes to userspace:
- Disable the udev rule for updating kdump on hot un/plug changes.
Add the following as the first two lines to the udev rule file
/usr/lib/udev/rules.d/98-kexec.rules:
# For x86_64, the kernel handles updates to crash elfcorehdr
CONST{arch}=="x86-64", GOTO="kdump_reload_end"
These two lines will cause cpu and memory hot un/plug events
to be skipped within this rule file, for x86_64.
- Change to the kexec_file_load for loading the kdump kernel:
Eg. on RHEL: in /usr/bin/kdumpctl, change to:
standard_kexec_args="-p -d -s"
which adds the -s to select kexec_file_load syscall.
This patchset supports kexec_load with a modified kexec userspace
utility, and a working changeset to the kexec userspace utility
is provided here (and to use, the above change to standard_kexec_args
would be, for example, to append --hotplug-size=131072 instead of -s).
diff --git a/kexec/arch/i386/crashdump-x86.c b/kexec/arch/i386/crashdump-x86.c
index 9826f6d..06adb7e 100644
--- a/kexec/arch/i386/crashdump-x86.c
+++ b/kexec/arch/i386/crashdump-x86.c
@@ -48,6 +48,7 @@
#include
extern struct arch_options_t arch_options;
+extern unsigned long long hotplug_size;
static int get_kernel_page_offset(struct kexec_info *UNUSED(info),
struct crash_elf_info *elf_info)
@@ -975,6 +976,13 @@ int load_crashdump_segments(struct kexec_info *info,
char* mod_cmdline,
} else {
memsz = bufsz;
}
+
+/* If hotplug support enabled, use that size */
+if (hotplug_size) {
+memsz = hotplug_size;
+}
+
+info->elfcorehdr =
elfcorehdr = add_buffer(info, tmp, bufsz, memsz, align, min_base,
max_addr, -1);
dbgprintf("Created elf header segment at 0x%lx\n", elfcorehdr);
diff --git a/kexec/kexec.c b/kexec/kexec.c
index f63b36b..9569d9a 100644
--- a/kexec/kexec.c
+++ b/kexec/kexec.c
@@ -58,6 +58,7 @@
unsigned long long mem_min = 0;
unsigned long long mem_max = ULONG_MAX;
+unsigned long long hotplug_size = 0;
static unsigned long kexec_flags = 0;
/* Flags for kexec file (fd) based syscall */
static unsigned long kexec_file_flags = 0;
@@ -672,6 +673,12 @@ static void update_purgatory(struct kexec_info *info)
if (info->segment[i].mem == (void *)info->rhdr.rel_addr) {
continue;
}
+/* Don't include elfcorehdr in the checksum, if hotplug
+ * support enabled.
+ */
+if (hotplug_size && (info->segment[i].mem == (void
*)info->elfcorehdr)) {
+ continue;
+ }
sha256_update(&ctx, info->segment[i].buf,
info->segment[i].bufsz);
nullsz = info->segment[i].memsz - info->segment[i].bufsz;
@@ -1504,6 +1511,17 @@ int main(int argc, char *argv[])
case OPT_PRINT_CKR_SIZE:
print_crashkernel_region_size();
return 0;
+ case OPT_HOTPLUG_SIZE:
+/* Reserved the specified size for hotplug growth */
+ hotplug_size = strtoul(optarg, &endptr, 0);
+ if (*endptr) {
+ fprintf(stderr,
+ "Bad option value in
--hotplug-size=%s\n",
+ optarg);
+ usage();
+ return 1;
+ }
+ break;
default:
break;
}
diff --git a/kexec/kexec.h b/kexec/kexec.h
index 595dd68..b30dda4 100644
--- a/kexec/kexec.h
+++ b/kexec/kexec.h
@@ -169,6 +169,7 @@ struct kexec_info {
int com
