Re: cleanup swiotlb initialization v8
On Mon, Apr 04, 2022 at 07:05:44AM +0200, Christoph Hellwig wrote: > Hi all, > > this series tries to clean up the swiotlb initialization, including > that of swiotlb-xen. To get there is also removes the x86 iommu table > infrastructure that massively obsfucates the initialization path. > > Git tree: > > git://git.infradead.org/users/hch/misc.git swiotlb-init-cleanup > > Gitweb: > > > http://git.infradead.org/users/hch/misc.git/shortlog/refs/heads/swiotlb-init-cleanup I've updated the git tree above with the commit message nitpicks and received reviews. I plan to pull the patches into the dma-mapping tree after -rc3 is released, so if any involved maintainer is not happy with the result, please speak up now.
Re: cleanup swiotlb initialization v8
On Mon, Apr 04, 2022 at 07:05:44AM +0200, Christoph Hellwig wrote: > Hi all, > > this series tries to clean up the swiotlb initialization, including > that of swiotlb-xen. To get there is also removes the x86 iommu table > infrastructure that massively obsfucates the initialization path. > > Git tree: > > git://git.infradead.org/users/hch/misc.git swiotlb-init-cleanup > > Gitweb: > > > http://git.infradead.org/users/hch/misc.git/shortlog/refs/heads/swiotlb-init-cleanup > > Changes since v7: > - rebased to Linux 5.18-rc1 > - better document the lower bound swiotlb size for xen-swiotlb > - improve the nslabs calculation for the retry case in >swiotlb_init_remap and swiotlb_init_late Hey Christoph, Feel free to tack on Reviewed-by: Konrad Rzeszutek Wilk on them if you would like. Thank you for doing the spring cleaning of this codebase!
Re: cleanup swiotlb initialization v8
On 4/4/22 1:05 AM, Christoph Hellwig wrote: Hi all, this series tries to clean up the swiotlb initialization, including that of swiotlb-xen. To get there is also removes the x86 iommu table infrastructure that massively obsfucates the initialization path. Git tree: git://git.infradead.org/users/hch/misc.git swiotlb-init-cleanup Gitweb: http://git.infradead.org/users/hch/misc.git/shortlog/refs/heads/swiotlb-init-cleanup Tested-by: Boris Ostrovsky
Re: cleanup swiotlb initialization
On 2/25/22 3:47 AM, Christoph Hellwig wrote: On Thu, Feb 24, 2022 at 12:07:26PM -0500, Boris Ostrovsky wrote: Just to be clear: this crashes only as dom0. Boots fine as baremetal. Ah. I can gues what this might be. On Xen the hypervisor controls the IOMMU and we should never end up initializing it in Linux, right? Right, we shouldn't be in that code path. Can you try the patch below on top of the series? Yes, this makes dom0 boot fine. (It also addresses something I wanted to mention while looking at the patches, which was to remove Xen initialization code from pci_swiotlb_detect_4gb() since it had noting to do with what the routine's name implies.) -boris
Re: cleanup swiotlb initialization
On Thu, Feb 24, 2022 at 12:07:26PM -0500, Boris Ostrovsky wrote:
>>> Just to be clear: this crashes only as dom0. Boots fine as baremetal.
>> Ah. I can gues what this might be. On Xen the hypervisor controls the
>> IOMMU and we should never end up initializing it in Linux, right?
>
>
> Right, we shouldn't be in that code path.
Can you try the patch below on top of the series?
diff --git a/arch/x86/kernel/pci-dma.c b/arch/x86/kernel/pci-dma.c
index b849f11a756d0..f88c9a4a5ed12 100644
--- a/arch/x86/kernel/pci-dma.c
+++ b/arch/x86/kernel/pci-dma.c
@@ -47,22 +47,6 @@ static unsigned int x86_swiotlb_flags;
*/
static void __init pci_swiotlb_detect_4gb(void)
{
-#ifdef CONFIG_SWIOTLB_XEN
- if (xen_pv_domain()) {
- if (xen_initial_domain())
- x86_swiotlb_enable = true;
-
- if (x86_swiotlb_enable) {
- dma_ops = &xen_swiotlb_dma_ops;
-#ifdef CONFIG_PCI
- /* Make sure ACS will be enabled */
- pci_request_acs();
-#endif
- }
- return;
- }
-#endif /* CONFIG_SWIOTLB_XEN */
-
/* don't initialize swiotlb if iommu=off (no_iommu=1) */
if (!no_iommu && max_possible_pfn > MAX_DMA32_PFN)
x86_swiotlb_enable = true;
@@ -82,16 +66,36 @@ static inline void __init pci_swiotlb_detect_4gb(void)
}
#endif /* CONFIG_SWIOTLB */
+#ifdef CONFIG_SWIOTLB_XEN
+static void __init xen_swiotlb_init(void)
+{
+ if (!xen_initial_domain() && !x86_swiotlb_enable)
+ return;
+ x86_swiotlb_enable = true;
+ dma_ops = &xen_swiotlb_dma_ops;
+#ifdef CONFIG_PCI
+ /* Make sure ACS will be enabled */
+ pci_request_acs();
+#endif
+ swiotlb_init_remap(true, x86_swiotlb_flags, xen_swiotlb_fixup);
+}
+#else
+static inline void __init xen_swiotlb_init(void)
+{
+}
+#endif /* CONFIG_SWIOTLB_XEN */
+
void __init pci_iommu_alloc(void)
{
+ if (xen_pv_domain()) {
+ xen_swiotlb_init();
+ return;
+ }
pci_swiotlb_detect_4gb();
gart_iommu_hole_init();
amd_iommu_detect();
detect_intel_iommu();
-#ifdef CONFIG_SWIOTLB
- swiotlb_init_remap(x86_swiotlb_enable, x86_swiotlb_flags,
- xen_pv_domain() ? xen_swiotlb_fixup : NULL);
-#endif
+ swiotlb_init(x86_swiotlb_enable, x86_swiotlb_flags);
}
/*
Re: cleanup swiotlb initialization
On 2/24/22 11:39 AM, Christoph Hellwig wrote: On Thu, Feb 24, 2022 at 11:18:33AM -0500, Boris Ostrovsky wrote: On 2/24/22 10:58 AM, Christoph Hellwig wrote: Thanks. This looks really strange as early_amd_iommu_init should not interact much with the changes. I'll see if I can find a AMD system to test on. Just to be clear: this crashes only as dom0. Boots fine as baremetal. Ah. I can gues what this might be. On Xen the hypervisor controls the IOMMU and we should never end up initializing it in Linux, right? Right, we shouldn't be in that code path. -boris
Re: cleanup swiotlb initialization
On Thu, Feb 24, 2022 at 11:18:33AM -0500, Boris Ostrovsky wrote: > > On 2/24/22 10:58 AM, Christoph Hellwig wrote: >> Thanks. >> >> This looks really strange as early_amd_iommu_init should not interact much >> with the changes. I'll see if I can find a AMD system to test on. > > > Just to be clear: this crashes only as dom0. Boots fine as baremetal. Ah. I can gues what this might be. On Xen the hypervisor controls the IOMMU and we should never end up initializing it in Linux, right?
Re: cleanup swiotlb initialization
On 2/24/22 10:58 AM, Christoph Hellwig wrote: Thanks. This looks really strange as early_amd_iommu_init should not interact much with the changes. I'll see if I can find a AMD system to test on. Just to be clear: this crashes only as dom0. Boots fine as baremetal. -boris On Wed, Feb 23, 2022 at 07:57:49PM -0500, Boris Ostrovsky wrote: [ 37.377313] BUG: unable to handle page fault for address: c90042880018 [ 37.378219] #PF: supervisor read access in kernel mode [ 37.378219] #PF: error_code(0x) - not-present page [ 37.378219] PGD 7c2f2ee067 P4D 7c2f2ee067 PUD 7bf019b067 PMD 105a30067 PTE 0 [ 37.378219] Oops: [#1] PREEMPT SMP NOPTI [ 37.378219] CPU: 14 PID: 1 Comm: swapper/0 Not tainted 5.17.0-rc5swiotlb #9 [ 37.378219] Hardware name: Oracle Corporation ORACLE SERVER E1-2c/ASY,Generic,SM,E1-2c, BIOS 49004900 12/23/2020 [ 37.378219] RIP: e030:init_iommu_one+0x248/0x2f0 [ 37.378219] Code: 48 89 43 68 48 85 c0 74 c4 be 00 20 00 00 48 89 df e8 ea ee ff ff 48 89 43 78 48 85 c0 74 ae c6 83 98 00 00 00 00 48 8b 43 38 <48> 8b 40 18 a8 01 74 07 83 8b a8 04 00 00 01 f6 83 a8 04 00 00 01 [ 37.378219] RSP: e02b:c9004044bd18 EFLAGS: 00010286 [ 37.378219] RAX: c9004288 RBX: 888107260800 RCX: [ 37.378219] RDX: 8000 RSI: ea00041cab80 RDI: [ 37.378219] RBP: c9004044bd38 R08: 0901 R09: ea00041cab00 [ 37.378219] R10: 0002 R11: R12: c90040435008 [ 37.378219] R13: 0008 R14: efa0 R15: [ 37.378219] FS: () GS:88fef418() knlGS: [ 37.378219] CS: e030 DS: ES: CR0: 80050033 [ 37.378219] CR2: c90042880018 CR3: 0260a000 CR4: 00050660 [ 37.378219] Call Trace: [ 37.378219] [ 37.378219] early_amd_iommu_init+0x3c5/0x72d [ 37.378219] ? iommu_setup+0x284/0x284 [ 37.378219] state_next+0x158/0x68f [ 37.378219] ? iommu_setup+0x284/0x284 [ 37.378219] iommu_go_to_state+0x28/0x2d [ 37.378219] amd_iommu_init+0x15/0x4b [ 37.378219] ? iommu_setup+0x284/0x284 [ 37.378219] pci_iommu_init+0x12/0x37 [ 37.378219] do_one_initcall+0x48/0x210 [ 37.378219] kernel_init_freeable+0x229/0x28c [ 37.378219] ? rest_init+0xe0/0xe0 [ 37.963966] kernel_init+0x1a/0x130 [ 37.979415] ret_from_fork+0x22/0x30 [ 37.991436] [ 37.999465] Modules linked in: [ 38.007413] CR2: c90042880018 [ 38.019416] ---[ end trace ]--- [ 38.023418] RIP: e030:init_iommu_one+0x248/0x2f0 [ 38.023418] Code: 48 89 43 68 48 85 c0 74 c4 be 00 20 00 00 48 89 df e8 ea ee ff ff 48 89 43 78 48 85 c0 74 ae c6 83 98 00 00 00 00 48 8b 43 38 <48> 8b 40 18 a8 01 74 07 83 8b a8 04 00 00 01 f6 83 a8 04 00 00 01 [ 38.023418] RSP: e02b:c9004044bd18 EFLAGS: 00010286 [ 38.023418] RAX: c9004288 RBX: 888107260800 RCX: [ 38.155413] RDX: 8000 RSI: ea00041cab80 RDI: [ 38.175965] Freeing initrd memory: 62640K [ 38.155413] RBP: c9004044bd38 R08: 0901 R09: ea00041cab00 [ 38.155413] R10: 0002 R11: R12: c90040435008 [ 38.155413] R13: 0008 R14: efa0 R15: [ 38.155413] FS: () GS:88fef418() knlGS: [ 38.287414] CS: e030 DS: ES: CR0: 80050033 [ 38.309557] CR2: c90042880018 CR3: 0260a000 CR4: 00050660 [ 38.332403] Kernel panic - not syncing: Fatal exception [ 38.351414] Rebooting in 20 seconds.. -boris ---end quoted text---
Re: cleanup swiotlb initialization
Thanks. This looks really strange as early_amd_iommu_init should not interact much with the changes. I'll see if I can find a AMD system to test on. On Wed, Feb 23, 2022 at 07:57:49PM -0500, Boris Ostrovsky wrote: > [ 37.377313] BUG: unable to handle page fault for address: c90042880018 > [ 37.378219] #PF: supervisor read access in kernel mode > [ 37.378219] #PF: error_code(0x) - not-present page > [ 37.378219] PGD 7c2f2ee067 P4D 7c2f2ee067 PUD 7bf019b067 PMD 105a30067 PTE > 0 > [ 37.378219] Oops: [#1] PREEMPT SMP NOPTI > [ 37.378219] CPU: 14 PID: 1 Comm: swapper/0 Not tainted 5.17.0-rc5swiotlb #9 > [ 37.378219] Hardware name: Oracle Corporation ORACLE SERVER > E1-2c/ASY,Generic,SM,E1-2c, BIOS 49004900 12/23/2020 > [ 37.378219] RIP: e030:init_iommu_one+0x248/0x2f0 > [ 37.378219] Code: 48 89 43 68 48 85 c0 74 c4 be 00 20 00 00 48 89 df e8 ea > ee ff ff 48 89 43 78 48 85 c0 74 ae c6 83 98 00 00 00 00 48 8b 43 38 <48> 8b > 40 18 a8 01 74 07 83 8b a8 04 00 00 01 f6 83 a8 04 00 00 01 > [ 37.378219] RSP: e02b:c9004044bd18 EFLAGS: 00010286 > [ 37.378219] RAX: c9004288 RBX: 888107260800 RCX: > > [ 37.378219] RDX: 8000 RSI: ea00041cab80 RDI: > > [ 37.378219] RBP: c9004044bd38 R08: 0901 R09: > ea00041cab00 > [ 37.378219] R10: 0002 R11: R12: > c90040435008 > [ 37.378219] R13: 0008 R14: efa0 R15: > > [ 37.378219] FS: () GS:88fef418() > knlGS: > [ 37.378219] CS: e030 DS: ES: CR0: 80050033 > [ 37.378219] CR2: c90042880018 CR3: 0260a000 CR4: > 00050660 > [ 37.378219] Call Trace: > [ 37.378219] > [ 37.378219] early_amd_iommu_init+0x3c5/0x72d > [ 37.378219] ? iommu_setup+0x284/0x284 > [ 37.378219] state_next+0x158/0x68f > [ 37.378219] ? iommu_setup+0x284/0x284 > [ 37.378219] iommu_go_to_state+0x28/0x2d > [ 37.378219] amd_iommu_init+0x15/0x4b > [ 37.378219] ? iommu_setup+0x284/0x284 > [ 37.378219] pci_iommu_init+0x12/0x37 > [ 37.378219] do_one_initcall+0x48/0x210 > [ 37.378219] kernel_init_freeable+0x229/0x28c > [ 37.378219] ? rest_init+0xe0/0xe0 > [ 37.963966] kernel_init+0x1a/0x130 > [ 37.979415] ret_from_fork+0x22/0x30 > [ 37.991436] > [ 37.999465] Modules linked in: > [ 38.007413] CR2: c90042880018 > [ 38.019416] ---[ end trace ]--- > [ 38.023418] RIP: e030:init_iommu_one+0x248/0x2f0 > [ 38.023418] Code: 48 89 43 68 48 85 c0 74 c4 be 00 20 00 00 48 89 df e8 ea > ee ff ff 48 89 43 78 48 85 c0 74 ae c6 83 98 00 00 00 00 48 8b 43 38 <48> 8b > 40 18 a8 01 74 07 83 8b a8 04 00 00 01 f6 83 a8 04 00 00 01 > [ 38.023418] RSP: e02b:c9004044bd18 EFLAGS: 00010286 > [ 38.023418] RAX: c9004288 RBX: 888107260800 RCX: > > [ 38.155413] RDX: 8000 RSI: ea00041cab80 RDI: > > [ 38.175965] Freeing initrd memory: 62640K > [ 38.155413] RBP: c9004044bd38 R08: 0901 R09: > ea00041cab00 > [ 38.155413] R10: 0002 R11: R12: > c90040435008 > [ 38.155413] R13: 0008 R14: efa0 R15: > > [ 38.155413] FS: () GS:88fef418() > knlGS: > [ 38.287414] CS: e030 DS: ES: CR0: 80050033 > [ 38.309557] CR2: c90042880018 CR3: 0260a000 CR4: > 00050660 > [ 38.332403] Kernel panic - not syncing: Fatal exception > [ 38.351414] Rebooting in 20 seconds.. > > > > -boris ---end quoted text---
Re: cleanup swiotlb initialization
On 2/22/22 10:35 AM, Christoph Hellwig wrote: Hi all, this series tries to clean up the swiotlb initialization, including that of swiotlb-xen. To get there is also removes the x86 iommu table infrastructure that massively obsfucates the initialization path. Git tree: git://git.infradead.org/users/hch/misc.git swiotlb-init-cleanup I haven't had a chance to look at this yet but this crashes as dom0: [ 37.377313] BUG: unable to handle page fault for address: c90042880018 [ 37.378219] #PF: supervisor read access in kernel mode [ 37.378219] #PF: error_code(0x) - not-present page [ 37.378219] PGD 7c2f2ee067 P4D 7c2f2ee067 PUD 7bf019b067 PMD 105a30067 PTE 0 [ 37.378219] Oops: [#1] PREEMPT SMP NOPTI [ 37.378219] CPU: 14 PID: 1 Comm: swapper/0 Not tainted 5.17.0-rc5swiotlb #9 [ 37.378219] Hardware name: Oracle Corporation ORACLE SERVER E1-2c/ASY,Generic,SM,E1-2c, BIOS 49004900 12/23/2020 [ 37.378219] RIP: e030:init_iommu_one+0x248/0x2f0 [ 37.378219] Code: 48 89 43 68 48 85 c0 74 c4 be 00 20 00 00 48 89 df e8 ea ee ff ff 48 89 43 78 48 85 c0 74 ae c6 83 98 00 00 00 00 48 8b 43 38 <48> 8b 40 18 a8 01 74 07 83 8b a8 04 00 00 01 f6 83 a8 04 00 00 01 [ 37.378219] RSP: e02b:c9004044bd18 EFLAGS: 00010286 [ 37.378219] RAX: c9004288 RBX: 888107260800 RCX: [ 37.378219] RDX: 8000 RSI: ea00041cab80 RDI: [ 37.378219] RBP: c9004044bd38 R08: 0901 R09: ea00041cab00 [ 37.378219] R10: 0002 R11: R12: c90040435008 [ 37.378219] R13: 0008 R14: efa0 R15: [ 37.378219] FS: () GS:88fef418() knlGS: [ 37.378219] CS: e030 DS: ES: CR0: 80050033 [ 37.378219] CR2: c90042880018 CR3: 0260a000 CR4: 00050660 [ 37.378219] Call Trace: [ 37.378219] [ 37.378219] early_amd_iommu_init+0x3c5/0x72d [ 37.378219] ? iommu_setup+0x284/0x284 [ 37.378219] state_next+0x158/0x68f [ 37.378219] ? iommu_setup+0x284/0x284 [ 37.378219] iommu_go_to_state+0x28/0x2d [ 37.378219] amd_iommu_init+0x15/0x4b [ 37.378219] ? iommu_setup+0x284/0x284 [ 37.378219] pci_iommu_init+0x12/0x37 [ 37.378219] do_one_initcall+0x48/0x210 [ 37.378219] kernel_init_freeable+0x229/0x28c [ 37.378219] ? rest_init+0xe0/0xe0 [ 37.963966] kernel_init+0x1a/0x130 [ 37.979415] ret_from_fork+0x22/0x30 [ 37.991436] [ 37.999465] Modules linked in: [ 38.007413] CR2: c90042880018 [ 38.019416] ---[ end trace ]--- [ 38.023418] RIP: e030:init_iommu_one+0x248/0x2f0 [ 38.023418] Code: 48 89 43 68 48 85 c0 74 c4 be 00 20 00 00 48 89 df e8 ea ee ff ff 48 89 43 78 48 85 c0 74 ae c6 83 98 00 00 00 00 48 8b 43 38 <48> 8b 40 18 a8 01 74 07 83 8b a8 04 00 00 01 f6 83 a8 04 00 00 01 [ 38.023418] RSP: e02b:c9004044bd18 EFLAGS: 00010286 [ 38.023418] RAX: c9004288 RBX: 888107260800 RCX: [ 38.155413] RDX: 8000 RSI: ea00041cab80 RDI: [ 38.175965] Freeing initrd memory: 62640K [ 38.155413] RBP: c9004044bd38 R08: 0901 R09: ea00041cab00 [ 38.155413] R10: 0002 R11: R12: c90040435008 [ 38.155413] R13: 0008 R14: efa0 R15: [ 38.155413] FS: () GS:88fef418() knlGS: [ 38.287414] CS: e030 DS: ES: CR0: 80050033 [ 38.309557] CR2: c90042880018 CR3: 0260a000 CR4: 00050660 [ 38.332403] Kernel panic - not syncing: Fatal exception [ 38.351414] Rebooting in 20 seconds.. -boris
