On 07.08.24 12:23, Marek Marczykowski-Górecki wrote:
On Tue, Aug 06, 2024 at 05:24:22PM +0200, Jürgen Groß wrote:
On 06.08.24 17:21, Marek Marczykowski-Górecki wrote:
On Tue, Aug 06, 2024 at 04:12:32PM +0200, Jürgen Groß wrote:
Marek,
On 17.06.24 16:03, Marek Marczykowski-Górecki wrote:
On Mon, Jun 17, 2024 at 01:22:37PM +0200, Jan Beulich wrote:
Hello,
while it feels like we had a similar situation before, I can't seem to be
able to find traces thereof, or associated (Linux) commits.
Is it some AMD Threadripper system by a chance? Previous thread on this
issue:
https://lore.kernel.org/xen-devel/CAOCpoWdOH=xgxiqsc1c5ueb1thxajh4wizbczq-qt+d_kak...@mail.gmail.com/
With
(XEN) Dom0 kernel: 64-bit, PAE, lsb, paddr 0x1000000 -> 0x4000000
...
(XEN) Dom0 alloc.: 0000000440000000->0000000448000000 (619175 pages to be
allocated)
...
(XEN) Loaded kernel: ffffffff81000000->ffffffff84000000
the kernel occupies the space from 16Mb to 64Mb in the initial allocation.
Page tables come (almost) directly above:
(XEN) Page tables: ffffffff84001000->ffffffff84026000
I.e. they're just above the 64Mb boundary. Yet sadly in the host E820 map
there is
(XEN) [0000000004000000, 0000000004009fff] (ACPI NVS)
i.e. a non-RAM range starting at 64Mb. The kernel (currently) won't tolerate
such an overlap (also if it was overlapping the kernel image, e.g. if on the
machine in question s sufficiently much larger kernel was used). Yet with its
fundamental goal of making its E820 match the host one I'm also in trouble
thinking of possible solutions / workarounds. I certainly do not see Xen
trying to cover for this, as the E820 map re-arrangement is purely a kernel
side decision (forward ported kernels got away without, and what e.g. the
BSDs do is entirely unknown to me).
In Qubes we have worked around the issue by moving the kernel lower
(CONFIG_PHYSICAL_START=0x200000):
https://github.com/QubesOS/qubes-linux-kernel/commit/3e8be4ac1682370977d4d0dc1d782c428d860282
Far from ideal, but gets it bootable...
could you test the attached kernel patches? They should fix the issue without
having to modify CONFIG_PHYSICAL_START.
I have tested them to boot up without problem on my test system, but I don't
have access to a system showing the E820 map conflict you are seeing.
The patches have been developed against kernel 6.11-rc2, but I think they
should apply to a 6.10 and maybe even an older kernel.
Sure, but tomorrow-ish.
Thanks.
Seems to work :)
Snippets from Xen log:
(XEN) EFI RAM map:
(XEN) [0000000000000000, 000000000009ffff] (usable)
(XEN) [00000000000a0000, 00000000000fffff] (reserved)
(XEN) [0000000000100000, 0000000003ffffff] (usable)
(XEN) [0000000004000000, 0000000004011fff] (ACPI NVS)
(XEN) [0000000004012000, 0000000009df1fff] (usable)
(XEN) [0000000009df2000, 0000000009ffffff] (reserved)
(XEN) [000000000a000000, 00000000a8840fff] (usable)
(XEN) [00000000a8841000, 00000000a9d9ffff] (reserved)
(XEN) [00000000a9da0000, 00000000a9dd4fff] (ACPI data)
(XEN) [00000000a9dd5000, 00000000a9dd5fff] (reserved)
(XEN) [00000000a9dd6000, 00000000a9f20fff] (ACPI data)
(XEN) [00000000a9f21000, 00000000aa099fff] (ACPI NVS)
(XEN) [00000000aa09a000, 00000000ab1fefff] (reserved)
(XEN) [00000000ab1ff000, 00000000abffffff] (usable)
(XEN) [00000000ac000000, 00000000afffffff] (reserved)
(XEN) [00000000b2500000, 00000000b2580fff] (reserved)
(XEN) [00000000b3580000, 00000000b3600fff] (reserved)
(XEN) [00000000e2100000, 00000000e2280fff] (reserved)
(XEN) [00000000fa180000, 00000000fa200fff] (reserved)
(XEN) [00000000fa300000, 00000000fa3fffff] (reserved)
(XEN) [00000000fea00000, 00000000feafffff] (reserved)
(XEN) [00000000fec00000, 00000000fec00fff] (reserved)
(XEN) [00000000fec10000, 00000000fec10fff] (reserved)
(XEN) [00000000fed00000, 00000000fed00fff] (reserved)
(XEN) [00000000fed40000, 00000000fed44fff] (reserved)
(XEN) [00000000fed80000, 00000000fed8ffff] (reserved)
(XEN) [00000000fedc2000, 00000000fedcffff] (reserved)
(XEN) [00000000fedd4000, 00000000fedd5fff] (reserved)
(XEN) [00000000fee00000, 00000000feefffff] (reserved)
(XEN) [00000000ff000000, 00000000ffffffff] (reserved)
(XEN) [0000000100000000, 000000104f1fffff] (usable)
(XEN) [000000104f200000, 000000104fffffff] (reserved)
(XEN) [0000010000000000, 00000100103fffff] (reserved)
(XEN) [0000018030000000, 00000180403fffff] (reserved)
(XEN) [0000018060000000, 00000180703fffff] (reserved)
(XEN) [0000020090000000, 00000200a03fffff] (reserved)
...
(XEN) Dom0 has maximum 1400 PIRQs
(XEN) Xen kernel: 64-bit, lsb
(XEN) Dom0 kernel: 64-bit, PAE, lsb, paddr 0x1000000 -> 0x4800000
(XEN) PHYSICAL MEMORY ARRANGEMENT:
(XEN) Dom0 alloc.: 0000001010000000->0000001018000000 (1000315 pages to
be allocated)
(XEN) Init. ramdisk: 000000104b57b000->000000104f1ffe72
(XEN) VIRTUAL MEMORY ARRANGEMENT:
(XEN) Loaded kernel: ffffffff81000000->ffffffff84800000
(XEN) Phys-Mach map: 0000008000000000->0000008000800000
(XEN) Start info: ffffffff84800000->ffffffff848004b8
(XEN) Page tables: ffffffff84801000->ffffffff8482a000
(XEN) Boot stack: ffffffff8482a000->ffffffff8482b000
(XEN) TOTAL: ffffffff80000000->ffffffff84c00000
(XEN) ENTRY ADDRESS: ffffffff838b7640
So, it would indeed conflict with the ACPI NVS region, but the system
started, and later dom0 reports this region remapped:
[ 0.000000] BIOS-provided physical RAM map:
[ 0.000000] Xen: [mem 0x0000000000000000-0x000000000007ffff] usable
[ 0.000000] Xen: [mem 0x0000000000080000-0x00000000000fffff] reserved
[ 0.000000] Xen: [mem 0x0000000000100000-0x0000000009df1fff] usable
[ 0.000000] Xen: [mem 0x0000000009df2000-0x0000000009ffffff] reserved
[ 0.000000] Xen: [mem 0x000000000a000000-0x00000000a8840fff] usable
[ 0.000000] Xen: [mem 0x00000000a8841000-0x00000000a9d9ffff] reserved
[ 0.000000] Xen: [mem 0x00000000a9da0000-0x00000000a9dd4fff] ACPI data
[ 0.000000] Xen: [mem 0x00000000a9dd5000-0x00000000a9dd5fff] reserved
[ 0.000000] Xen: [mem 0x00000000a9dd6000-0x00000000a9f20fff] ACPI data
[ 0.000000] Xen: [mem 0x00000000a9f21000-0x00000000aa099fff] ACPI NVS
[ 0.000000] Xen: [mem 0x00000000aa09a000-0x00000000ab1fefff] reserved
[ 0.000000] Xen: [mem 0x00000000ab1ff000-0x00000000abffffff] usable
[ 0.000000] Xen: [mem 0x00000000ac000000-0x00000000afffffff] reserved
[ 0.000000] Xen: [mem 0x00000000b2500000-0x00000000b2580fff] reserved
[ 0.000000] Xen: [mem 0x00000000b3580000-0x00000000b3600fff] reserved
[ 0.000000] Xen: [mem 0x00000000e2100000-0x00000000e2280fff] reserved
[ 0.000000] Xen: [mem 0x00000000fa180000-0x00000000fa200fff] reserved
[ 0.000000] Xen: [mem 0x00000000fa300000-0x00000000fa3fffff] reserved
[ 0.000000] Xen: [mem 0x00000000fea00000-0x00000000feafffff] reserved
[ 0.000000] Xen: [mem 0x00000000fec00000-0x00000000fec00fff] reserved
[ 0.000000] Xen: [mem 0x00000000fec10000-0x00000000fec10fff] reserved
[ 0.000000] Xen: [mem 0x00000000fed00000-0x00000000fed00fff] reserved
[ 0.000000] Xen: [mem 0x00000000fed40000-0x00000000fed44fff] reserved
[ 0.000000] Xen: [mem 0x00000000fed80000-0x00000000fed8ffff] reserved
[ 0.000000] Xen: [mem 0x00000000fedc2000-0x00000000fedcffff] reserved
[ 0.000000] Xen: [mem 0x00000000fedd4000-0x00000000fedd5fff] reserved
[ 0.000000] Xen: [mem 0x00000000fee00000-0x00000000feefffff] reserved
[ 0.000000] Xen: [mem 0x00000000ff000000-0x00000000ffffffff] reserved
[ 0.000000] Xen: [mem 0x0000000100000000-0x0000000156c4bfff] usable
[ 0.000000] Xen: [mem 0x000000104f1ee000-0x000000104f1fffff] ACPI NVS
[ 0.000000] Xen: [mem 0x000000104f200000-0x000000104fffffff] reserved
[ 0.000000] Xen: [mem 0x0000010000000000-0x00000100103fffff] reserved
[ 0.000000] Xen: [mem 0x0000018030000000-0x00000180403fffff] reserved
[ 0.000000] Xen: [mem 0x0000018060000000-0x00000180703fffff] reserved
[ 0.000000] Xen: [mem 0x0000020090000000-0x00000200a03fffff] reserved
Thanks for the testing!
Are you fine with me adding your "Tested-by:"?
Juergen
