Re: [PATCH 2/2] powerpc/tpm: Reserve SML log when kexec'ing
Stefan Berger writes: > On 2/23/23 22:25, Michael Ellerman wrote: >> The TPM code in prom_init.c creates a small buffer of memory to store >> the TPM's SML (Stored Measurement Log). It's communicated to Linux via >> the linux,sml-base/size device tree properties of the TPM node. >> >> When kexec'ing that buffer can be overwritten, or when kdump'ing it may >> not be mapped by the second kernel. The latter can lead to a crash when >> booting the second kernel such as: >> >>tpm_ibmvtpm 7103: CRQ initialization completed >>BUG: Unable to handle kernel data access on read at 0xc0002ffb >>Faulting instruction address: 0xc000200a70e0 >>Oops: Kernel access of bad area, sig: 11 [#1] >>LE PAGE_SIZE=64K MMU=Radix SMP NR_CPUS=2048 NUMA pSeries >>Modules linked in: >>CPU: 0 PID: 1 Comm: swapper/0 Not tainted 6.2.0-rc2-00134-g9307ce092f5d >> #314 >>Hardware name: IBM pSeries (emulated by qemu) POWER9 (raw) 0x4e1200 >> 0xf05 of:SLOF,git-5b4c5a pSeries >>NIP: c000200a70e0 LR: c000203dd5dc CTR: 0800 >>REGS: c00024543280 TRAP: 0300 Not tainted >> (6.2.0-rc2-00134-g9307ce092f5d) >>MSR: 82009033 CR: 24002280 XER: >> 0006 >>CFAR: c000200a70c8 DAR: c0002ffb DSISR: 4000 IRQMASK: 0 >>... >>NIP memcpy_power7+0x400/0x7d0 >>LR kmemdup+0x5c/0x80 >>Call Trace: >> memcpy_power7+0x274/0x7d0 (unreliable) >> kmemdup+0x5c/0x80 >> tpm_read_log_of+0xe8/0x1b0 >> tpm_bios_log_setup+0x60/0x210 >> tpm_chip_register+0x134/0x320 >> tpm_ibmvtpm_probe+0x520/0x7d0 >> vio_bus_probe+0x9c/0x460 >> really_probe+0x104/0x420 >> __driver_probe_device+0xb0/0x170 >> driver_probe_device+0x58/0x180 >> __driver_attach+0xd8/0x250 >> bus_for_each_dev+0xb4/0x140 >> driver_attach+0x34/0x50 >> bus_add_driver+0x1e8/0x2d0 >> driver_register+0xb4/0x1c0 >> __vio_register_driver+0x74/0x9c >> ibmvtpm_module_init+0x34/0x48 >> do_one_initcall+0x80/0x320 >> kernel_init_freeable+0x304/0x3ac >> kernel_init+0x30/0x1a0 >> ret_from_kernel_thread+0x5c/0x64 > > I have not been able to reproduce this particular crash issue with a > 6.2 kernel running on P10 PowerVM when NOT applying your patches. The crash only happens for a crashdump kernel, not a regular kexec. And depending on where the SML is in memory, compared to where the crashkernel is, the SML might be mapped accidentally in which case there is no crash. > For my tests I have used the following parameter with the 16GB VM: > crashkernel=2G-4G:384M,4G-16G:1G,16G-64G:2G,64G-128G:2G,128G-:4G So you should be seeing a 2GB crashkernel reservation at 512MB. > What I noticed is that the log gets corrupted when the 2 patches are applied: > > After fresh boot: > >> cp /sys/kernel/security/tpm0/binary_bios_measurements ./ >> ls -l binary_bios_measurements > -r--r-. 1 root root 10051 Feb 28 12:09 binary_bios_measurements > > >> kexec -l /boot/vmlinuz-6.2.0+ --initrd /boot/initramfs-6.2.0+.img >> '--append=BOOT_IMAGE=/vmlinuz-6.2.0+ root=/dev/mapper/rhel_XYZ ro >> crashkernel=2G-4G:384M,4G-16G:1G,16G-64G:2G,64G-128G:2G,128G-:4G >> rd.lvm.lv=rhel_XYZ/root rd.lvm.lv=rhel_XYZ/swap biosdevname=0' -s >> kexec -e That's a normal kexec, not a crash kexec, so it doesn't use the crashkernel region mentioned above. >> cp /sys/kernel/security/tpm0/binary_bios_measurements ./ >> ls -l binary_bios_measurements > -r--r-. 1 root root 32 Feb 28 12:10 binary_bios_measurements > >> od -t x1 < binary_bios_measurements > 000 d0 0d fe ed 00 00 77 80 00 00 00 a0 00 00 4f 4c > 020 00 00 00 28 00 00 00 11 00 00 00 11 00 00 00 00 > 040 That's a device tree header !? O_o #define OF_DT_HEADER0xd00dfeed /* marker */ > The contents have changed and these first 4 bytes of it are always the > same once it has become this 32 byte file, otherwise they would be > zero. I'm not sure what's happening there. We'll need to debug it some more :/ cheers
Re: [PATCH 2/2] powerpc/tpm: Reserve SML log when kexec'ing
On 2/23/23 22:25, Michael Ellerman wrote: The TPM code in prom_init.c creates a small buffer of memory to store the TPM's SML (Stored Measurement Log). It's communicated to Linux via the linux,sml-base/size device tree properties of the TPM node. When kexec'ing that buffer can be overwritten, or when kdump'ing it may not be mapped by the second kernel. The latter can lead to a crash when booting the second kernel such as: tpm_ibmvtpm 7103: CRQ initialization completed BUG: Unable to handle kernel data access on read at 0xc0002ffb Faulting instruction address: 0xc000200a70e0 Oops: Kernel access of bad area, sig: 11 [#1] LE PAGE_SIZE=64K MMU=Radix SMP NR_CPUS=2048 NUMA pSeries Modules linked in: CPU: 0 PID: 1 Comm: swapper/0 Not tainted 6.2.0-rc2-00134-g9307ce092f5d #314 Hardware name: IBM pSeries (emulated by qemu) POWER9 (raw) 0x4e1200 0xf05 of:SLOF,git-5b4c5a pSeries NIP: c000200a70e0 LR: c000203dd5dc CTR: 0800 REGS: c00024543280 TRAP: 0300 Not tainted (6.2.0-rc2-00134-g9307ce092f5d) MSR: 82009033 CR: 24002280 XER: 0006 CFAR: c000200a70c8 DAR: c0002ffb DSISR: 4000 IRQMASK: 0 ... NIP memcpy_power7+0x400/0x7d0 LR kmemdup+0x5c/0x80 Call Trace: memcpy_power7+0x274/0x7d0 (unreliable) kmemdup+0x5c/0x80 tpm_read_log_of+0xe8/0x1b0 tpm_bios_log_setup+0x60/0x210 tpm_chip_register+0x134/0x320 tpm_ibmvtpm_probe+0x520/0x7d0 vio_bus_probe+0x9c/0x460 really_probe+0x104/0x420 __driver_probe_device+0xb0/0x170 driver_probe_device+0x58/0x180 __driver_attach+0xd8/0x250 bus_for_each_dev+0xb4/0x140 driver_attach+0x34/0x50 bus_add_driver+0x1e8/0x2d0 driver_register+0xb4/0x1c0 __vio_register_driver+0x74/0x9c ibmvtpm_module_init+0x34/0x48 do_one_initcall+0x80/0x320 kernel_init_freeable+0x304/0x3ac kernel_init+0x30/0x1a0 ret_from_kernel_thread+0x5c/0x64 I have not been able to reproduce this particular crash issue with a 6.2 kernel running on P10 PowerVM when NOT applying your patches. For my tests I have used the following parameter with the 16GB VM: crashkernel=2G-4G:384M,4G-16G:1G,16G-64G:2G,64G-128G:2G,128G-:4G What I noticed is that the log gets corrupted when the 2 patches are applied: After fresh boot: cp /sys/kernel/security/tpm0/binary_bios_measurements ./ ls -l binary_bios_measurements -r--r-. 1 root root 10051 Feb 28 12:09 binary_bios_measurements kexec -l /boot/vmlinuz-6.2.0+ --initrd /boot/initramfs-6.2.0+.img '--append=BOOT_IMAGE=/vmlinuz-6.2.0+ root=/dev/mapper/rhel_XYZ ro crashkernel=2G-4G:384M,4G-16G:1G,16G-64G:2G,64G-128G:2G,128G-:4G rd.lvm.lv=rhel_XYZ/root rd.lvm.lv=rhel_XYZ/swap biosdevname=0' -s kexec -e cp /sys/kernel/security/tpm0/binary_bios_measurements ./ ls -l binary_bios_measurements -r--r-. 1 root root 32 Feb 28 12:10 binary_bios_measurements od -t x1 < binary_bios_measurements 000 d0 0d fe ed 00 00 77 80 00 00 00 a0 00 00 4f 4c 020 00 00 00 28 00 00 00 11 00 00 00 11 00 00 00 00 040 The contents have changed and these first 4 bytes of it are always the same once it has become this 32 byte file, otherwise they would be zero. The address and size parameters passed around in this patch seem good, though. Stefan
[PATCH 2/2] powerpc/tpm: Reserve SML log when kexec'ing
The TPM code in prom_init.c creates a small buffer of memory to store
the TPM's SML (Stored Measurement Log). It's communicated to Linux via
the linux,sml-base/size device tree properties of the TPM node.
When kexec'ing that buffer can be overwritten, or when kdump'ing it may
not be mapped by the second kernel. The latter can lead to a crash when
booting the second kernel such as:
tpm_ibmvtpm 7103: CRQ initialization completed
BUG: Unable to handle kernel data access on read at 0xc0002ffb
Faulting instruction address: 0xc000200a70e0
Oops: Kernel access of bad area, sig: 11 [#1]
LE PAGE_SIZE=64K MMU=Radix SMP NR_CPUS=2048 NUMA pSeries
Modules linked in:
CPU: 0 PID: 1 Comm: swapper/0 Not tainted 6.2.0-rc2-00134-g9307ce092f5d #314
Hardware name: IBM pSeries (emulated by qemu) POWER9 (raw) 0x4e1200 0xf05
of:SLOF,git-5b4c5a pSeries
NIP: c000200a70e0 LR: c000203dd5dc CTR: 0800
REGS: c00024543280 TRAP: 0300 Not tainted
(6.2.0-rc2-00134-g9307ce092f5d)
MSR: 82009033 CR: 24002280 XER: 0006
CFAR: c000200a70c8 DAR: c0002ffb DSISR: 4000 IRQMASK: 0
...
NIP memcpy_power7+0x400/0x7d0
LR kmemdup+0x5c/0x80
Call Trace:
memcpy_power7+0x274/0x7d0 (unreliable)
kmemdup+0x5c/0x80
tpm_read_log_of+0xe8/0x1b0
tpm_bios_log_setup+0x60/0x210
tpm_chip_register+0x134/0x320
tpm_ibmvtpm_probe+0x520/0x7d0
vio_bus_probe+0x9c/0x460
really_probe+0x104/0x420
__driver_probe_device+0xb0/0x170
driver_probe_device+0x58/0x180
__driver_attach+0xd8/0x250
bus_for_each_dev+0xb4/0x140
driver_attach+0x34/0x50
bus_add_driver+0x1e8/0x2d0
driver_register+0xb4/0x1c0
__vio_register_driver+0x74/0x9c
ibmvtpm_module_init+0x34/0x48
do_one_initcall+0x80/0x320
kernel_init_freeable+0x304/0x3ac
kernel_init+0x30/0x1a0
ret_from_kernel_thread+0x5c/0x64
To fix the crash, add the SML region to the usable memory areas for the
kdump kernel, so that the second kernel will map the region. To avoid
corruption of the region, add the region to the reserved memory areas,
so that the second kernel does not use the memory for something else.
Reported-by: Stefan Berger
Signed-off-by: Michael Ellerman
---
arch/powerpc/include/asm/kexec_ranges.h | 1 +
arch/powerpc/kexec/file_load_64.c | 12
arch/powerpc/kexec/ranges.c | 20
3 files changed, 33 insertions(+)
diff --git a/arch/powerpc/include/asm/kexec_ranges.h
b/arch/powerpc/include/asm/kexec_ranges.h
index f83866a19e87..cf6a97f9113d 100644
--- a/arch/powerpc/include/asm/kexec_ranges.h
+++ b/arch/powerpc/include/asm/kexec_ranges.h
@@ -21,5 +21,6 @@ int add_kernel_mem_range(struct crash_mem **mem_ranges);
int add_rtas_mem_range(struct crash_mem **mem_ranges);
int add_opal_mem_range(struct crash_mem **mem_ranges);
int add_reserved_mem_ranges(struct crash_mem **mem_ranges);
+int add_sml_mem_range(struct crash_mem **mem_ranges);
#endif /* _ASM_POWERPC_KEXEC_RANGES_H */
diff --git a/arch/powerpc/kexec/file_load_64.c
b/arch/powerpc/kexec/file_load_64.c
index 110d28bede2a..90c10a89fcbc 100644
--- a/arch/powerpc/kexec/file_load_64.c
+++ b/arch/powerpc/kexec/file_load_64.c
@@ -79,6 +79,10 @@ static int get_exclude_memory_ranges(struct crash_mem
**mem_ranges)
if (ret)
goto out;
+ ret = add_sml_mem_range(mem_ranges);
+ if (ret)
+ goto out;
+
ret = add_opal_mem_range(mem_ranges);
if (ret)
goto out;
@@ -122,6 +126,10 @@ static int get_usable_memory_ranges(struct crash_mem
**mem_ranges)
if (ret)
goto out;
+ ret = add_sml_mem_range(mem_ranges);
+ if (ret)
+ goto out;
+
ret = add_opal_mem_range(mem_ranges);
if (ret)
goto out;
@@ -225,6 +233,10 @@ static int get_reserved_memory_ranges(struct crash_mem
**mem_ranges)
if (ret)
goto out;
+ ret = add_sml_mem_range(mem_ranges);
+ if (ret)
+ goto out;
+
ret = add_tce_mem_ranges(mem_ranges);
if (ret)
goto out;
diff --git a/arch/powerpc/kexec/ranges.c b/arch/powerpc/kexec/ranges.c
index 5fc53a5fcfdf..8b01655ceb5e 100644
--- a/arch/powerpc/kexec/ranges.c
+++ b/arch/powerpc/kexec/ranges.c
@@ -350,6 +350,26 @@ int add_rtas_mem_range(struct crash_mem **mem_ranges)
return ret;
}
+int add_sml_mem_range(struct crash_mem **mem_ranges)
+{
+ struct device_node *dn;
+ int ret = 0;
+ u64 base;
+ u32 size;
+
+ // Matches the device type in tpm_ibmvtpm.c
+ for_each_node_by_type(dn, "IBM,vtpm") {
+ if (of_property_read_u64(dn, "linux,sml-base", ) == 0 &&
+ of_property_read_u32(dn, "linux,sml-size", ) == 0) {
+ ret = add_mem_range(mem_ranges, base, size);
+ if (ret)
+
