Re: Please test: UVM fault unlocking (aka vmobjlock)
On Tue, Dec 07, 2021 at 10:04:04AM -0600, Scott Cheloha wrote: > On Mon, Nov 29, 2021 at 10:50:32PM +0100, Martin Pieuchot wrote: > > On 24/11/21(Wed) 11:16, Martin Pieuchot wrote: > > > Diff below unlock the bottom part of the UVM fault handler. I'm > > > interested in squashing the remaining bugs. Please test with your usual > > > setup & report back. > > > > Thanks to all the testers, here's a new version that includes a bug fix. > > > > Tests on !x86 architectures are much appreciated! > > [...] > > witness: lock order reversal: > 1st 0xfd83d37b23c8 uobjlk (>vmobjlock) > 2nd 0x815fdb00 objmm (>mm.lock) > lock order ">mm.lock"(rwlock) -> ">vmobjlock"(rwlock) first seen > at: > #0 rw_enter+0x68 > #1 uvm_obj_wire+0x4f > #2 shmem_get_pages+0xae > #3 __i915_gem_object_get_pages+0x85 > #4 i915_vma_pin_ww+0x451 > #5 i915_ggtt_pin+0x61 > #6 intel_execlists_submission_setup+0x396 > #7 intel_engines_init+0x2ff > #8 intel_gt_init+0x136 > #9 i915_gem_init+0x9d > #10 i915_driver_probe+0x760 > #11 inteldrm_attachhook+0x46 > #12 config_process_deferred_mountroot+0x5b > #13 main+0x743 > lock order ">vmobjlock"(rwlock) -> ">mm.lock"(rwlock) first seen > at: > #0 rw_enter+0x68 > #1 __i915_gem_object_get_pages+0x29 > #2 i915_gem_fault+0x1cb > #3 drm_fault+0x163 > #4 uvm_fault+0x19b > #5 upageflttrap+0x5e > #6 usertrap+0x190 > #7 recall_trap+0x8 Seeing the same reversal with the same stack trace during all boots. Also, just saw a real novelty, a simultaneous trap panic and a warning printf from vref(). The console showed this: pWaAnRiNcI:NG: vSrPeLf OTs eLdO ER e Dr eO Nv gTeRtA Pr eEqXuIiTr eda 0 Which I deciphered into: WARNING: vref used where vget required and panic: SPL NOT LOWERED ON TRAP EXIT a 0 The warning printf is from vref() in vfs_subr.c. The panic could be from Xintr_user_exit() in locore.S or alltraps_kern() in vector.S. Not familiar enough with that code to say which. The console locked up and I was not able to look at anything in ddb. Same machine as before: > dmesg: > > OpenBSD 7.0-current (GENERIC.MP) #17: Tue Dec 7 09:39:06 CST 2021 > ssc@jetsam.local:/usr/src/sys/arch/amd64/compile/GENERIC.MP > real mem = 16895528960 (16112MB) > avail mem = 16237621248 (15485MB) > random: good seed from bootblocks > mpath0 at root > scsibus0 at mpath0: 256 targets > mainbus0 at root > bios0 at mainbus0: SMBIOS rev. 3.0 @ 0x9f03b000 (63 entries) > bios0: vendor LENOVO version "N23ET59W (1.34 )" date 11/08/2018 > bios0: LENOVO 20KHCTO1WW > acpi0 at bios0: ACPI 5.0 > acpi0: sleep states S0 S3 S4 S5 > acpi0: tables DSDT FACP SSDT SSDT TPM2 UEFI SSDT SSDT HPET APIC MCFG ECDT > SSDT SSDT BOOT BATB SSDT SSDT SSDT LPIT WSMT SSDT SSDT SSDT DBGP DBG2 MSDM > DMAR NHLT ASF! FPDT UEFI > acpi0: wakeup devices GLAN(S4) XHC_(S3) XDCI(S4) HDAS(S4) RP01(S4) PXSX(S4) > RP02(S4) PXSX(S4) PXSX(S4) RP04(S4) PXSX(S4) RP05(S4) PXSX(S4) RP06(S4) > PXSX(S4) RP07(S4) [...] > acpitimer0 at acpi0: 3579545 Hz, 24 bits > acpihpet0 at acpi0: 2399 Hz > acpimadt0 at acpi0 addr 0xfee0: PC-AT compat > cpu0 at mainbus0: apid 0 (boot processor) > cpu0: Intel(R) Core(TM) i7-8650U CPU @ 1.90GHz, 1795.82 MHz, 06-8e-0a > cpu0: > FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,SRBDS_CTRL,MD_CLEAR,TSXFA,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN > cpu0: 256KB 64b/line 8-way L2 cache > cpu0: smt 0, core 0, package 0 > mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges > cpu0: apic clock running at 24MHz > cpu0: mwait min=64, max=64, C-substates=0.2.1.2.4.1.1.1, IBE > cpu1 at mainbus0: apid 2 (application processor) > tsc: cpu0/cpu1 sync round 1: 1865 regressions > tsc: cpu0 lags cpu1 by 0 cycles > tsc: cpu1 lags cpu0 by 56 cycles > cpu1: Intel(R) Core(TM) i7-8650U CPU @ 1.90GHz, 1795.82 MHz, 06-8e-0a > cpu1: > FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,SRBDS_CTRL,MD_CLEAR,TSXFA,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN > cpu1: 256KB 64b/line 8-way L2 cache > cpu1: smt 0, core 1, package 0 > cpu2 at mainbus0: apid 4 (application processor) > tsc: cpu0/cpu2 sync round 1: 1921 regressions > tsc: cpu0 lags cpu2 by 0 cycles > tsc: cpu2 lags cpu0 by 60
Re: Please test: UVM fault unlocking (aka vmobjlock)
On Mon, Nov 29, 2021 at 10:50:32PM +0100, Martin Pieuchot wrote: > On 24/11/21(Wed) 11:16, Martin Pieuchot wrote: > > Diff below unlock the bottom part of the UVM fault handler. I'm > > interested in squashing the remaining bugs. Please test with your usual > > setup & report back. > > Thanks to all the testers, here's a new version that includes a bug fix. > > Tests on !x86 architectures are much appreciated! At the risk of not being appreciated, with this patch I see this reversal during boot: witness: lock order reversal: 1st 0xfd83d37b23c8 uobjlk (>vmobjlock) 2nd 0x815fdb00 objmm (>mm.lock) lock order ">mm.lock"(rwlock) -> ">vmobjlock"(rwlock) first seen at: #0 rw_enter+0x68 #1 uvm_obj_wire+0x4f #2 shmem_get_pages+0xae #3 __i915_gem_object_get_pages+0x85 #4 i915_vma_pin_ww+0x451 #5 i915_ggtt_pin+0x61 #6 intel_execlists_submission_setup+0x396 #7 intel_engines_init+0x2ff #8 intel_gt_init+0x136 #9 i915_gem_init+0x9d #10 i915_driver_probe+0x760 #11 inteldrm_attachhook+0x46 #12 config_process_deferred_mountroot+0x5b #13 main+0x743 lock order ">vmobjlock"(rwlock) -> ">mm.lock"(rwlock) first seen at: #0 rw_enter+0x68 #1 __i915_gem_object_get_pages+0x29 #2 i915_gem_fault+0x1cb #3 drm_fault+0x163 #4 uvm_fault+0x19b #5 upageflttrap+0x5e #6 usertrap+0x190 #7 recall_trap+0x8 dmesg: OpenBSD 7.0-current (GENERIC.MP) #17: Tue Dec 7 09:39:06 CST 2021 ssc@jetsam.local:/usr/src/sys/arch/amd64/compile/GENERIC.MP real mem = 16895528960 (16112MB) avail mem = 16237621248 (15485MB) random: good seed from bootblocks mpath0 at root scsibus0 at mpath0: 256 targets mainbus0 at root bios0 at mainbus0: SMBIOS rev. 3.0 @ 0x9f03b000 (63 entries) bios0: vendor LENOVO version "N23ET59W (1.34 )" date 11/08/2018 bios0: LENOVO 20KHCTO1WW acpi0 at bios0: ACPI 5.0 acpi0: sleep states S0 S3 S4 S5 acpi0: tables DSDT FACP SSDT SSDT TPM2 UEFI SSDT SSDT HPET APIC MCFG ECDT SSDT SSDT BOOT BATB SSDT SSDT SSDT LPIT WSMT SSDT SSDT SSDT DBGP DBG2 MSDM DMAR NHLT ASF! FPDT UEFI acpi0: wakeup devices GLAN(S4) XHC_(S3) XDCI(S4) HDAS(S4) RP01(S4) PXSX(S4) RP02(S4) PXSX(S4) PXSX(S4) RP04(S4) PXSX(S4) RP05(S4) PXSX(S4) RP06(S4) PXSX(S4) RP07(S4) [...] acpitimer0 at acpi0: 3579545 Hz, 24 bits acpihpet0 at acpi0: 2399 Hz acpimadt0 at acpi0 addr 0xfee0: PC-AT compat cpu0 at mainbus0: apid 0 (boot processor) cpu0: Intel(R) Core(TM) i7-8650U CPU @ 1.90GHz, 1795.82 MHz, 06-8e-0a cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,SRBDS_CTRL,MD_CLEAR,TSXFA,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN cpu0: 256KB 64b/line 8-way L2 cache cpu0: smt 0, core 0, package 0 mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges cpu0: apic clock running at 24MHz cpu0: mwait min=64, max=64, C-substates=0.2.1.2.4.1.1.1, IBE cpu1 at mainbus0: apid 2 (application processor) tsc: cpu0/cpu1 sync round 1: 1865 regressions tsc: cpu0 lags cpu1 by 0 cycles tsc: cpu1 lags cpu0 by 56 cycles cpu1: Intel(R) Core(TM) i7-8650U CPU @ 1.90GHz, 1795.82 MHz, 06-8e-0a cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,SRBDS_CTRL,MD_CLEAR,TSXFA,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN cpu1: 256KB 64b/line 8-way L2 cache cpu1: smt 0, core 1, package 0 cpu2 at mainbus0: apid 4 (application processor) tsc: cpu0/cpu2 sync round 1: 1921 regressions tsc: cpu0 lags cpu2 by 0 cycles tsc: cpu2 lags cpu0 by 60 cycles cpu2: Intel(R) Core(TM) i7-8650U CPU @ 1.90GHz, 1794.85 MHz, 06-8e-0a cpu2: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,SRBDS_CTRL,MD_CLEAR,TSXFA,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN cpu2: 256KB 64b/line 8-way L2 cache cpu2: smt 0, core 2, package 0 cpu3 at mainbus0: apid 6 (application processor) tsc: cpu0/cpu3 sync round 1: 1819 regressions tsc: cpu0 lags cpu3 by 0 cycles tsc: cpu3 lags cpu0 by 60 cycles
Re: Please test: UVM fault unlocking (aka vmobjlock)
On Dec 04 14:28:23, h...@stare.cz wrote: > On Dec 04 12:23:42, h...@stare.cz wrote: > > On Dec 01 09:07:43, h...@stare.cz wrote: > > > > > > On 24/11/21(Wed) 11:16, Martin Pieuchot wrote: > > > > > > > Diff below unlock the bottom part of the UVM fault handler. I'm > > > > > > > interested in squashing the remaining bugs. Please test with > > > > > > > your usual > > > > > > > setup & report back. > > > > > > > > > > > > Thanks to all the testers, here's a new version that includes a bug > > > > > > fix. > > > > > > Tests on !x86 architectures are much appreciated! current/amd64 on a Thinkpad T410 (dmesg below) having this patch just panicked with the following message: uvm_fault(0xfd8135c51178, 0x48, 0, 1) -> e I don't know if it's related to the patch, but given that the panic is uvm_faiult() ... Upon boot, the kernel said Dec 5 17:00:34 t410 /bsd: witness: lock order reversal: Dec 5 17:00:34 t410 /bsd: 1st 0xfd8115667438 uobjlk (>vmobjlock) Dec 5 17:00:34 t410 /bsd: 2nd 0x80bcf700 objmm (>mm.lock) Dec 5 17:00:34 t410 /bsd: lock order ">mm.lock"(rwlock) -> ">vmobjlock"(rwlock) first seen at: Dec 5 17:00:34 t410 /bsd: #0 rw_enter+0x65 Dec 5 17:00:34 t410 /bsd: #1 uvm_obj_wire+0x46 Dec 5 17:00:34 t410 /bsd: #2 shmem_get_pages+0xaf Dec 5 17:00:34 t410 /bsd: #3 __i915_gem_object_get_pages+0x9d Dec 5 17:00:34 t410 /bsd: #4 i915_vma_pin_ww+0x49b Dec 5 17:00:34 t410 /bsd: #5 i915_ggtt_pin+0x61 Dec 5 17:00:34 t410 /bsd: #6 __intel_context_do_pin_ww+0x2d1 Dec 5 17:00:34 t410 /bsd: #7 __intel_context_do_pin+0x4b Dec 5 17:00:34 t410 /bsd: #8 intel_engines_init+0x4a1 Dec 5 17:00:34 t410 /bsd: #9 intel_gt_init+0x133 Dec 5 17:00:34 t410 /bsd: #10 i915_gem_init+0xa3 Dec 5 17:00:34 t410 /bsd: #11 i915_driver_probe+0x75a Dec 5 17:00:34 t410 /bsd: #12 inteldrm_attachhook+0x45 Dec 5 17:00:34 t410 /bsd: #13 config_process_deferred_mountroot+0x6b Dec 5 17:00:34 t410 /bsd: #14 main+0x743 Dec 5 17:00:34 t410 /bsd: lock order ">vmobjlock"(rwlock) -> ">mm.lock"(rwlock) first seen at: Dec 5 17:00:34 t410 /bsd: #0 rw_enter+0x65 Dec 5 17:00:34 t410 /bsd: #1 __i915_gem_object_get_pages+0x30 Dec 5 17:00:34 t410 /bsd: #2 i915_gem_fault+0x408 Dec 5 17:00:34 t410 /bsd: #3 drm_fault+0x156 Dec 5 17:00:34 t410 /bsd: #4 uvm_fault+0x179 Dec 5 17:00:34 t410 /bsd: #5 upageflttrap+0x62 Dec 5 17:00:34 t410 /bsd: #6 usertrap+0x18b Dec 5 17:00:34 t410 /bsd: #7 recall_trap+0x8 Dec 5 17:00:34 t410 /bsd: witness: lock order reversal: Dec 5 17:00:34 t410 /bsd: 1st 0xfd8115667438 uobjlk (>vmobjlock) Dec 5 17:00:34 t410 /bsd: 2nd 0x80bd0e48 vmapg (>pages_mutex) Dec 5 17:00:34 t410 /bsd: lock order data w2 -> w1 missing Dec 5 17:00:34 t410 /bsd: lock order ">vmobjlock"(rwlock) -> ">pages_mutex"(rwlock) first seen at: Dec 5 17:00:34 t410 /bsd: #0 rw_enter+0x65 Dec 5 17:00:34 t410 /bsd: #1 i915_vma_pin_ww+0x1c7 Dec 5 17:00:34 t410 /bsd: #2 i915_gem_object_ggtt_pin_ww+0x222 Dec 5 17:00:34 t410 /bsd: #3 i915_gem_fault+0x450 Dec 5 17:00:34 t410 /bsd: #4 drm_fault+0x156 Dec 5 17:00:34 t410 /bsd: #5 uvm_fault+0x179 Dec 5 17:00:34 t410 /bsd: #6 upageflttrap+0x62 Dec 5 17:00:34 t410 /bsd: #7 usertrap+0x18b Dec 5 17:00:34 t410 /bsd: #8 recall_trap+0x8 Dec 5 17:00:34 t410 /bsd: witness: lock order reversal: Dec 5 17:00:34 t410 /bsd: 1st 0xfd8115667438 uobjlk (>vmobjlock) Dec 5 17:00:34 t410 /bsd: 2nd 0x80111070 vmlk (>mutex) Dec 5 17:00:34 t410 /bsd: lock order data w2 -> w1 missing Dec 5 17:00:34 t410 /bsd: lock order ">vmobjlock"(rwlock) -> ">mutex"(rwlock) first seen at: Dec 5 17:00:34 t410 /bsd: #0 rw_enter+0x65 Dec 5 17:00:34 t410 /bsd: #1 i915_vma_pin_ww+0x2f2 Dec 5 17:00:34 t410 /bsd: #2 i915_gem_object_ggtt_pin_ww+0x222 Dec 5 17:00:34 t410 /bsd: #3 i915_gem_fault+0x450 Dec 5 17:00:34 t410 /bsd: #4 drm_fault+0x156 Dec 5 17:00:34 t410 /bsd: #5 uvm_fault+0x179 Dec 5 17:00:34 t410 /bsd: #6 upageflttrap+0x62 Dec 5 17:00:34 t410 /bsd: #7 usertrap+0x18b Dec 5 17:00:34 t410 /bsd: #8 recall_trap+0x8 but the machine was running as usual, with about three chrome tabs, and mplayer playing some audio files. Then, after about two hours: Dec 5 18:46:36 t410 /bsd: uvm_fault(0xfd8135c51178, 0x48, 0, 1) -> e Dec 5 18:46:36 t410 /bsd: kernel: page fault trap, code=0 Dec 5 18:46:36 t410 /bsd: Stopped at pledge_recvfd+0x59: movl 0x48(%rsi),%eax Dec 5 18:46:36 t410 /bsd: TIDPIDUID PRFLAGS PFLAGS CPU COMMAND Dec 5 18:46:36 t410 /bsd: * 13684 98899 10000x33 0x4000K chrome Dec 5 18:46:36 t410 /bsd: 137636 44471 990x100010 02 sndiod Dec 5 18:46:36 t410 /bsd: pledge_recvfd(8000334e4a90,0) at pledge_recvfd+0x59 Dec 5 18:46:36 t410 /bsd: unp_externalize(fd8063543600,210,80) at unp_externalize+0x10c Dec 5 18:46:36 t410 /bsd:
Re: Please test: UVM fault unlocking (aka vmobjlock)
On Dec 04 12:23:42, h...@stare.cz wrote: > On Dec 01 09:07:43, h...@stare.cz wrote: > > > > > On 24/11/21(Wed) 11:16, Martin Pieuchot wrote: > > > > > > Diff below unlock the bottom part of the UVM fault handler. I'm > > > > > > interested in squashing the remaining bugs. Please test with your > > > > > > usual > > > > > > setup & report back. > > > > > > > > > > Thanks to all the testers, here's a new version that includes a bug > > > > > fix. > > > > > Tests on !x86 architectures are much appreciated! > > > > > > This is current/arm64 on a Thinkpad W500 (dmesg below), Sorry, current/amd64, obviously. > running with the patch and option WITNESS and kern.witness.watch=2. > Three reversals are reported after boot: > > witness: lock order reversal: > 1st 0xfd8217a167c0 uobjlk (>vmobjlock) > 2nd 0x813e1b00 objmm (>mm.lock) > lock order ">mm.lock"(rwlock) -> ">vmobjlock"(rwlock) first seen > at: > #0 rw_enter+0x65 > #1 uvm_obj_wire+0x46 > #2 shmem_get_pages+0xaf > #3 __i915_gem_object_get_pages+0x9d > #4 i915_vma_pin_ww+0x49b > #5 i915_ggtt_pin+0x61 > #6 __intel_context_do_pin_ww+0x2d1 > #7 __intel_context_do_pin+0x4b > #8 intel_engines_init+0x4a1 > #9 intel_gt_init+0x133 > #10 i915_gem_init+0xa3 > #11 i915_driver_probe+0x75a > #12 inteldrm_attachhook+0x45 > #13 config_process_deferred_mountroot+0x6b > #14 main+0x743 > lock order ">vmobjlock"(rwlock) -> ">mm.lock"(rwlock) first seen > at: > #0 rw_enter+0x65 > #1 __i915_gem_object_get_pages+0x30 > #2 i915_gem_fault+0x1aa > #3 drm_fault+0x156 > #4 uvm_fault+0x179 > #5 upageflttrap+0x62 > #6 usertrap+0x18b > #7 recall_trap+0x8 > > witness: lock order reversal: > 1st 0xfd8218c24a98 uobjlk (>vmobjlock) > 2nd 0x80117070 vmlk (>mutex) > lock order data w2 -> w1 missing > lock order ">vmobjlock"(rwlock) -> ">mutex"(rwlock) first seen at: > #0 rw_enter+0x65 > #1 i915_vma_pin_ww+0x2f2 > #2 i915_gem_object_ggtt_pin_ww+0x222 > #3 i915_gem_fault+0x450 > #4 drm_fault+0x156 > #5 uvm_fault+0x179 > #6 upageflttrap+0x62 > #7 usertrap+0x18b > #8 recall_trap+0x8 > > witness: lock order reversal: > 1st 0xfd8217731670 uobjlk (>vmobjlock) > 2nd 0x814154c8 vmapg (>pages_mutex) > lock order data w2 -> w1 missing > lock order ">vmobjlock"(rwlock) -> ">pages_mutex"(rwlock) first > seen at: > #0 rw_enter+0x65 > #1 i915_vma_pin_ww+0x1c7 > #2 i915_gem_object_ggtt_pin_ww+0x222 > #3 i915_gem_fault+0x450 > #4 drm_fault+0x156 > #5 uvm_fault+0x179 > #6 upageflttrap+0x62 > #7 usertrap+0x18b > #8 recall_trap+0x8 > > > Jan > > OpenBSD 7.0-current (GENERIC.MP) #0: Sat Dec 4 10:02:46 CET 2021 > h...@w500.stare.cz:/usr/src/sys/arch/amd64/compile/GENERIC.MP > real mem = 8463781888 (8071MB) > avail mem = 8122720256 (7746MB) > random: good seed from bootblocks > mpath0 at root > scsibus0 at mpath0: 256 targets > mainbus0 at root > bios0 at mainbus0: SMBIOS rev. 2.4 @ 0xe0010 (80 entries) > bios0: vendor LENOVO version "6FET79WW (3.09 )" date 10/02/2009 > bios0: LENOVO 40612JG > acpi0 at bios0: ACPI 3.0 > acpi0: sleep states S0 S3 S4 S5 > acpi0: tables DSDT FACP SSDT ECDT APIC MCFG HPET SLIC BOOT ASF! SSDT TCPA > SSDT SSDT SSDT > acpi0: wakeup devices LID_(S3) SLPB(S3) IGBE(S4) EXP0(S4) EXP1(S4) EXP2(S4) > EXP3(S4) EXP4(S4) PCI1(S4) USB0(S3) USB3(S3) USB5(S3) EHC0(S3) EHC1(S3) > HDEF(S4) > acpitimer0 at acpi0: 3579545 Hz, 24 bits > acpiec0 at acpi0 > acpimadt0 at acpi0 addr 0xfee0: PC-AT compat > cpu0 at mainbus0: apid 0 (boot processor) > cpu0: Intel(R) Core(TM)2 Duo CPU P9500 @ 2.53GHz, 2527.42 MHz, 06-17-06 > cpu0: > FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,NXE,LONG,LAHF,PERF,SENSOR,MELTDOWN > cpu0: 6MB 64b/line 16-way L2 cache > cpu0: smt 0, core 0, package 0 > mtrr: Pentium Pro MTRR support, 7 var ranges, 88 fixed ranges > cpu0: apic clock running at 265MHz > cpu0: mwait min=64, max=64, C-substates=0.2.2.2.2.1.3, IBE > cpu1 at mainbus0: apid 1 (application processor) > cpu1: Intel(R) Core(TM)2 Duo CPU P9500 @ 2.53GHz, 2527.01 MHz, 06-17-06 > cpu1: > FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,NXE,LONG,LAHF,PERF,SENSOR,MELTDOWN > cpu1: 6MB 64b/line 16-way L2 cache > cpu1: smt 0, core 1, package 0 > ioapic0 at mainbus0: apid 1 pa 0xfec0, version 20, 24 pins, remapped > acpimcfg0 at acpi0 > acpimcfg0: addr 0xe000, bus 0-63 > acpihpet0 at acpi0: 14318179 Hz > acpiprt0 at acpi0: bus 0 (PCI0) > acpiprt1 at acpi0: bus 1 (AGP_) > acpiprt2 at acpi0: bus 2 (EXP0) > acpiprt3 at acpi0: bus 3 (EXP1) > acpiprt4 at acpi0: bus -1 (EXP2) > acpiprt5 at acpi0: bus 5 (EXP3) > acpiprt6 at acpi0: bus 13 (EXP4) > acpiprt7 at acpi0: bus 21 (PCI1) > acpibtn0 at acpi0: LID_ > acpibtn1 at acpi0: SLPB > acpipci0 at
Re: Please test: UVM fault unlocking (aka vmobjlock)
On Dec 01 09:07:43, h...@stare.cz wrote: > > > > On 24/11/21(Wed) 11:16, Martin Pieuchot wrote: > > > > > Diff below unlock the bottom part of the UVM fault handler. I'm > > > > > interested in squashing the remaining bugs. Please test with your > > > > > usual > > > > > setup & report back. > > > > > > > > Thanks to all the testers, here's a new version that includes a bug fix. > > > > Tests on !x86 architectures are much appreciated! > > > This is current/arm64 on a Thinkpad W500 (dmesg below), running with the patch and option WITNESS and kern.witness.watch=2. Three reversals are reported after boot: witness: lock order reversal: 1st 0xfd8217a167c0 uobjlk (>vmobjlock) 2nd 0x813e1b00 objmm (>mm.lock) lock order ">mm.lock"(rwlock) -> ">vmobjlock"(rwlock) first seen at: #0 rw_enter+0x65 #1 uvm_obj_wire+0x46 #2 shmem_get_pages+0xaf #3 __i915_gem_object_get_pages+0x9d #4 i915_vma_pin_ww+0x49b #5 i915_ggtt_pin+0x61 #6 __intel_context_do_pin_ww+0x2d1 #7 __intel_context_do_pin+0x4b #8 intel_engines_init+0x4a1 #9 intel_gt_init+0x133 #10 i915_gem_init+0xa3 #11 i915_driver_probe+0x75a #12 inteldrm_attachhook+0x45 #13 config_process_deferred_mountroot+0x6b #14 main+0x743 lock order ">vmobjlock"(rwlock) -> ">mm.lock"(rwlock) first seen at: #0 rw_enter+0x65 #1 __i915_gem_object_get_pages+0x30 #2 i915_gem_fault+0x1aa #3 drm_fault+0x156 #4 uvm_fault+0x179 #5 upageflttrap+0x62 #6 usertrap+0x18b #7 recall_trap+0x8 witness: lock order reversal: 1st 0xfd8218c24a98 uobjlk (>vmobjlock) 2nd 0x80117070 vmlk (>mutex) lock order data w2 -> w1 missing lock order ">vmobjlock"(rwlock) -> ">mutex"(rwlock) first seen at: #0 rw_enter+0x65 #1 i915_vma_pin_ww+0x2f2 #2 i915_gem_object_ggtt_pin_ww+0x222 #3 i915_gem_fault+0x450 #4 drm_fault+0x156 #5 uvm_fault+0x179 #6 upageflttrap+0x62 #7 usertrap+0x18b #8 recall_trap+0x8 witness: lock order reversal: 1st 0xfd8217731670 uobjlk (>vmobjlock) 2nd 0x814154c8 vmapg (>pages_mutex) lock order data w2 -> w1 missing lock order ">vmobjlock"(rwlock) -> ">pages_mutex"(rwlock) first seen at: #0 rw_enter+0x65 #1 i915_vma_pin_ww+0x1c7 #2 i915_gem_object_ggtt_pin_ww+0x222 #3 i915_gem_fault+0x450 #4 drm_fault+0x156 #5 uvm_fault+0x179 #6 upageflttrap+0x62 #7 usertrap+0x18b #8 recall_trap+0x8 Jan OpenBSD 7.0-current (GENERIC.MP) #0: Sat Dec 4 10:02:46 CET 2021 h...@w500.stare.cz:/usr/src/sys/arch/amd64/compile/GENERIC.MP real mem = 8463781888 (8071MB) avail mem = 8122720256 (7746MB) random: good seed from bootblocks mpath0 at root scsibus0 at mpath0: 256 targets mainbus0 at root bios0 at mainbus0: SMBIOS rev. 2.4 @ 0xe0010 (80 entries) bios0: vendor LENOVO version "6FET79WW (3.09 )" date 10/02/2009 bios0: LENOVO 40612JG acpi0 at bios0: ACPI 3.0 acpi0: sleep states S0 S3 S4 S5 acpi0: tables DSDT FACP SSDT ECDT APIC MCFG HPET SLIC BOOT ASF! SSDT TCPA SSDT SSDT SSDT acpi0: wakeup devices LID_(S3) SLPB(S3) IGBE(S4) EXP0(S4) EXP1(S4) EXP2(S4) EXP3(S4) EXP4(S4) PCI1(S4) USB0(S3) USB3(S3) USB5(S3) EHC0(S3) EHC1(S3) HDEF(S4) acpitimer0 at acpi0: 3579545 Hz, 24 bits acpiec0 at acpi0 acpimadt0 at acpi0 addr 0xfee0: PC-AT compat cpu0 at mainbus0: apid 0 (boot processor) cpu0: Intel(R) Core(TM)2 Duo CPU P9500 @ 2.53GHz, 2527.42 MHz, 06-17-06 cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,NXE,LONG,LAHF,PERF,SENSOR,MELTDOWN cpu0: 6MB 64b/line 16-way L2 cache cpu0: smt 0, core 0, package 0 mtrr: Pentium Pro MTRR support, 7 var ranges, 88 fixed ranges cpu0: apic clock running at 265MHz cpu0: mwait min=64, max=64, C-substates=0.2.2.2.2.1.3, IBE cpu1 at mainbus0: apid 1 (application processor) cpu1: Intel(R) Core(TM)2 Duo CPU P9500 @ 2.53GHz, 2527.01 MHz, 06-17-06 cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,NXE,LONG,LAHF,PERF,SENSOR,MELTDOWN cpu1: 6MB 64b/line 16-way L2 cache cpu1: smt 0, core 1, package 0 ioapic0 at mainbus0: apid 1 pa 0xfec0, version 20, 24 pins, remapped acpimcfg0 at acpi0 acpimcfg0: addr 0xe000, bus 0-63 acpihpet0 at acpi0: 14318179 Hz acpiprt0 at acpi0: bus 0 (PCI0) acpiprt1 at acpi0: bus 1 (AGP_) acpiprt2 at acpi0: bus 2 (EXP0) acpiprt3 at acpi0: bus 3 (EXP1) acpiprt4 at acpi0: bus -1 (EXP2) acpiprt5 at acpi0: bus 5 (EXP3) acpiprt6 at acpi0: bus 13 (EXP4) acpiprt7 at acpi0: bus 21 (PCI1) acpibtn0 at acpi0: LID_ acpibtn1 at acpi0: SLPB acpipci0 at acpi0 PCI0: 0x 0x0011 0x0001 acpicmos0 at acpi0 acpibat0 at acpi0: BAT0 model "COMPATIBLE" serial 1388 type LION oem "SANYO" acpiac0 at acpi0: AC unit online acpithinkpad0 at acpi0: version 1.0 "PNP0C14" at acpi0 not configured acpicpu0 at acpi0: !C3(100@57 mwait.3@0x30), !C2(500@1 mwait.1@0x10), C1(1000@1 mwait.1), PSS acpicpu1 at
Re: Please test: UVM fault unlocking (aka vmobjlock)
Hi, I've tested the second diff (containing the bug fix and I get a lock order reversal on bootup. On the first patch I was getting the same lock order reversal while using X and sometimes (only sometimes) would crash X. witness: lock order reversal: 1st 0xfd83fef05248 uobjlk (>vmobjlock) 2nd 0x8119f700 objmm (>mm.lock) lock order ">mm.lock"(rwlock) -> ">vmobjlock"(rwlock) first seen at: #0 rw_enter+0x65 #1 uvm_obj_wire+0x46 #2 shmem_get_pages+0xaf #3 __i915_gem_object_get_pages+0x9d #4 i915_vma_pin_ww+0x49b #5 i915_ggtt_pin+0x61 #6 intel_execlists_submission_setup+0x3b2 #7 intel_engines_init+0x2ff #8 intel_gt_init+0x133 #9 i915_gem_init+0xa3 #10 i915_driver_probe+0x75a #11 inteldrm_attachhook+0x45 #12 config_process_deferred_mountroot+0x6b #13 main+0x743 lock order ">vmobjlock"(rwlock) -> ">mm.lock"(rwlock) first seen at: #0 rw_enter+0x65 #1 __i915_gem_object_get_pages+0x30 #2 i915_gem_fault+0x1aa #3 drm_fault+0x156 #4 uvm_fault+0x179 #5 upageflttrap+0x62 #6 usertrap+0x18b #7 recall_trap+0x8 Dmesg: OpenBSD 7.0-current (GENERIC.MP) #3: Thu Dec 2 18:38:43 GMT 2021 pertho@pertho.local:/usr/src/sys/arch/amd64/compile/GENERIC.MP real mem = 17040445440 (16251MB) avail mem = 16376905728 (15618MB) random: good seed from bootblocks mpath0 at root scsibus0 at mpath0: 256 targets mainbus0 at root bios0 at mainbus0: SMBIOS rev. 3.0 @ 0x7b288000 (41 entries) bios0: vendor American Megatrends Inc. version "1.05.07" date 09/29/2017 bios0: PC Specialist LTD N13xWU acpi0 at bios0: ACPI 6.0 acpi0: sleep states S0 S3 S4 S5 acpi0: tables DSDT FACP APIC FPDT FIDT MCFG SSDT SSDT HPET UEFI SSDT SSDT SSDT DBGP DBG2 DMAR BGRT ASF! WSMT acpi0: wakeup devices RP17(S4) PXSX(S4) RP18(S4) PXSX(S4) RP19(S4) PXSX(S4) RP20(S4) PXSX(S4) RP21(S4) PXSX(S4) RP22(S4) PXSX(S4) RP23(S4) PXSX(S4) RP24(S4) PXSX(S4) [...] acpitimer0 at acpi0: 3579545 Hz, 24 bits acpimadt0 at acpi0 addr 0xfee0: PC-AT compat cpu0 at mainbus0: apid 0 (boot processor) cpu0: Intel(R) Core(TM) i7-8550U CPU @ 1.80GHz, 1696.05 MHz, 06-8e-0a cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,SGX,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,SRBDS_CTRL,MD_CLEAR,TSXFA,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN cpu0: 256KB 64b/line 8-way L2 cache cpu0: smt 0, core 0, package 0 mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges cpu0: apic clock running at 24MHz cpu0: mwait min=64, max=64, C-substates=0.2.1.2.4.1.1.1, IBE cpu1 at mainbus0: apid 2 (application processor) cpu1: Intel(R) Core(TM) i7-8550U CPU @ 1.80GHz, 1696.05 MHz, 06-8e-0a cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,SGX,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,SRBDS_CTRL,MD_CLEAR,TSXFA,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN cpu1: 256KB 64b/line 8-way L2 cache cpu1: smt 0, core 1, package 0 cpu2 at mainbus0: apid 4 (application processor) cpu2: Intel(R) Core(TM) i7-8550U CPU @ 1.80GHz, 1696.05 MHz, 06-8e-0a cpu2: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,SGX,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,SRBDS_CTRL,MD_CLEAR,TSXFA,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN cpu2: 256KB 64b/line 8-way L2 cache cpu2: smt 0, core 2, package 0 cpu3 at mainbus0: apid 6 (application processor) cpu3: Intel(R) Core(TM) i7-8550U CPU @ 1.80GHz, 1696.05 MHz, 06-8e-0a cpu3: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,SGX,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,SRBDS_CTRL,MD_CLEAR,TSXFA,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN cpu3: 256KB 64b/line 8-way L2 cache cpu3: smt 0, core 3, package 0 cpu4 at mainbus0: apid 1 (application processor) cpu4: Intel(R) Core(TM)
Re: Please test: UVM fault unlocking (aka vmobjlock)
On Mon, Nov 29, 2021 at 10:50:32PM +0100, Martin Pieuchot wrote: > On 24/11/21(Wed) 11:16, Martin Pieuchot wrote: > > Diff below unlock the bottom part of the UVM fault handler. I'm > > interested in squashing the remaining bugs. Please test with your usual > > setup & report back. > > Thanks to all the testers, here's a new version that includes a bug fix. > > Tests on !x86 architectures are much appreciated! GENERIC.MP build becomes noticably faster on a sparc64 T4-2 LDOM: This diff survived a full regress as well as all kernel builds. I've run regress and builds in parallel each, i.e. one `make' per regress/ and sparc64/compile/ subdirectory.
Re: Please test: UVM fault unlocking (aka vmobjlock)
On Mon, Nov 29, 2021 at 10:50:32PM +0100, Martin Pieuchot wrote: > On 24/11/21(Wed) 11:16, Martin Pieuchot wrote: > > Diff below unlock the bottom part of the UVM fault handler. I'm > > interested in squashing the remaining bugs. Please test with your usual > > setup & report back. > > Thanks to all the testers, here's a new version that includes a bug fix. I removed the witness part and ran it through my perfomamnce tests. 4 core machine http://bluhm.genua.de/perform/results/2021-12-01T23:10:12Z/perform.html 8 core, 2 socket machine http://bluhm.genua.de/perform/results/2021-12-01T09:53:27Z/perform.html The best improvement is make -j8 bsd on the 8 core machine, 22% system time, 5% real time decrease. I see slowdown in network performance, but that may be within the variation I always see when kernel layout changes. Here you can see best how kernel lock goes away in usertrap. http://bluhm.genua.de/perform/results/2021-12-01T09:53:27Z/2021-12-01T00%3A00%3A00Z/btrace/time_-lp_make_-CGENERIC.MP_-j8_-s-btrace-kstack.0.svg http://bluhm.genua.de/perform/results/2021-12-01T09:53:27Z/patch-mpi-uvm-unlock-nowitness.0/btrace/time_-lp_make_-CGENERIC.MP_-j8_-s-btrace-kstack.0.svg On 4 core machine system time during kernel build is reduced by 6%. bluhm
Re: Please test: UVM fault unlocking (aka vmobjlock)
> * Martin Pieuchot wrote: > > On 24/11/21(Wed) 11:16, Martin Pieuchot wrote: > > > Diff below unlock the bottom part of the UVM fault handler. I'm > > > interested in squashing the remaining bugs. Please test with your usual > > > setup & report back. > > > > Thanks to all the testers, here's a new version that includes a bug fix. > Hello, I use the patch since yesterday and didn't notice any issue so far. The system feels smoother, but I didn't measure. Below my dmesg. Regards. prx OpenBSD 7.0-current (GENERIC.MP) #2: Tue Nov 30 15:30:32 CET 2021 p...@shaihulud.at.home:/sys/arch/amd64/compile/GENERIC.MP real mem = 8254926848 (7872MB) avail mem = 7921725440 (7554MB) random: good seed from bootblocks mpath0 at root scsibus0 at mpath0: 256 targets mainbus0 at root bios0 at mainbus0: SMBIOS rev. 2.7 @ 0xccbfd000 (64 entries) bios0: vendor LENOVO version "N10ET40W (1.19 )" date 12/08/2015 bios0: LENOVO 20CLS4WV08 acpi0 at bios0: ACPI 5.0 acpi0: sleep states S0 S3 S4 S5 acpi0: tables DSDT FACP SLIC ASF! HPET ECDT APIC MCFG SSDT SSDT SSDT SSDT SSDT SSDT SSDT SSDT SSDT PCCT SSDT TCPA SSDT UEFI MSDM BATB FPDT UEFI DMAR acpi0: wakeup devices LID_(S4) SLPB(S3) IGBE(S4) EXP2(S4) XHCI(S3) EHC1(S3) acpitimer0 at acpi0: 3579545 Hz, 24 bits acpihpet0 at acpi0: 14318179 Hz acpiec0 at acpi0 acpimadt0 at acpi0 addr 0xfee0: PC-AT compat cpu0 at mainbus0: apid 0 (boot processor) cpu0: Intel(R) Core(TM) i5-5300U CPU @ 2.30GHz, 2195.21 MHz, 06-3d-04 cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,RDSEED,ADX,SMAP,PT,SRBDS_CTRL,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN cpu0: 256KB 64b/line 8-way L2 cache cpu0: smt 0, core 0, package 0 mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges cpu0: apic clock running at 99MHz cpu0: mwait min=64, max=64, C-substates=0.2.1.2.4.1.1.1, IBE cpu1 at mainbus0: apid 1 (application processor) cpu1: Intel(R) Core(TM) i5-5300U CPU @ 2.30GHz, 2194.92 MHz, 06-3d-04 cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,RDSEED,ADX,SMAP,PT,SRBDS_CTRL,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN cpu1: 256KB 64b/line 8-way L2 cache cpu1: smt 1, core 0, package 0 cpu2 at mainbus0: apid 2 (application processor) cpu2: Intel(R) Core(TM) i5-5300U CPU @ 2.30GHz, 2194.93 MHz, 06-3d-04 cpu2: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,RDSEED,ADX,SMAP,PT,SRBDS_CTRL,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN cpu2: 256KB 64b/line 8-way L2 cache cpu2: smt 0, core 1, package 0 cpu3 at mainbus0: apid 3 (application processor) cpu3: Intel(R) Core(TM) i5-5300U CPU @ 2.30GHz, 2194.92 MHz, 06-3d-04 cpu3: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,RDSEED,ADX,SMAP,PT,SRBDS_CTRL,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN cpu3: 256KB 64b/line 8-way L2 cache cpu3: smt 1, core 1, package 0 ioapic0 at mainbus0: apid 2 pa 0xfec0, version 20, 40 pins acpimcfg0 at acpi0 acpimcfg0: addr 0xf800, bus 0-63 acpiprt0 at acpi0: bus 0 (PCI0) acpiprt1 at acpi0: bus -1 (PEG_) acpiprt2 at acpi0: bus 2 (EXP1) acpiprt3 at acpi0: bus 3 (EXP2) acpiprt4 at acpi0: bus -1 (EXP3) acpibtn0 at acpi0: LID_ acpibtn1 at acpi0: SLPB acpipci0 at acpi0 PCI0: 0x 0x0011 0x0001 acpicmos0 at acpi0 acpibat0 at acpi0: BAT0 not present acpibat1 at acpi0: BAT1 model "45N1775" serial 315 type LION oem "SANYO" acpiac0 at acpi0: AC unit online acpithinkpad0 at acpi0: version 1.0 tpm0 at acpi0 TPM_ 1.2 (TIS) addr 0xfed4/0x5000, device 0x104a rev 0x4e "PNP0C14" at acpi0 not configured "PNP0C14" at acpi0 not configured "PNP0C14" at acpi0 not configured "INT340F" at acpi0 not configured
Re: Please test: UVM fault unlocking (aka vmobjlock)
Hi Martin, * Martin Pieuchot wrote: > On 24/11/21(Wed) 11:16, Martin Pieuchot wrote: > > Diff below unlock the bottom part of the UVM fault handler. I'm > > interested in squashing the remaining bugs. Please test with your usual > > setup & report back. > > Thanks to all the testers, here's a new version that includes a bug fix. As with the last patch, this one also works fine on my amd64 system. dmesg in my last message in the thread. Cheers Matthias
Re: Please test: UVM fault unlocking (aka vmobjlock)
On Mon, Nov 29, 2021 at 10:50:32PM +0100, Martin Pieuchot wrote: > On 24/11/21(Wed) 11:16, Martin Pieuchot wrote: > > Diff below unlock the bottom part of the UVM fault handler. I'm > > interested in squashing the remaining bugs. Please test with your usual > > setup & report back. > > Thanks to all the testers, here's a new version that includes a bug fix. > > Tests on !x86 architectures are much appreciated! Passed full regress on powerpc64.
Re: Please test: UVM fault unlocking (aka vmobjlock)
> > > On 24/11/21(Wed) 11:16, Martin Pieuchot wrote: > > > > Diff below unlock the bottom part of the UVM fault handler. I'm > > > > interested in squashing the remaining bugs. Please test with your usual > > > > setup & report back. > > > > > > Thanks to all the testers, here's a new version that includes a bug fix. > > > Tests on !x86 architectures are much appreciated! > > This is current/arm64 on a RPI4 (dmesg below), running with the patch and option WITNESS and kern.witness.watch=2; it has been recompiling the kernel for about 12 hours without reporting anything. Jan OpenBSD 7.0-current (GENERIC.MP) #0: Tue Nov 30 16:06:48 CET 2021 h...@pi.stare.cz:/usr/src/sys/arch/arm64/compile/GENERIC.MP real mem = 8419885056 (8029MB) avail mem = 8059887616 (7686MB) random: good seed from bootblocks mainbus0 at root: Raspberry Pi 4 Model B Rev 1.4 cpu0 at mainbus0 mpidr 0: ARM Cortex-A72 r0p3 cpu0: 48KB 64b/line 3-way L1 PIPT I-cache, 32KB 64b/line 2-way L1 D-cache cpu0: 1024KB 64b/line 16-way L2 cache cpu0: CRC32,ASID16 cpu1 at mainbus0 mpidr 1: ARM Cortex-A72 r0p3 cpu1: 48KB 64b/line 3-way L1 PIPT I-cache, 32KB 64b/line 2-way L1 D-cache cpu1: 1024KB 64b/line 16-way L2 cache cpu1: CRC32,ASID16 cpu2 at mainbus0 mpidr 2: ARM Cortex-A72 r0p3 cpu2: 48KB 64b/line 3-way L1 PIPT I-cache, 32KB 64b/line 2-way L1 D-cache cpu2: 1024KB 64b/line 16-way L2 cache cpu2: CRC32,ASID16 cpu3 at mainbus0 mpidr 3: ARM Cortex-A72 r0p3 cpu3: 48KB 64b/line 3-way L1 PIPT I-cache, 32KB 64b/line 2-way L1 D-cache cpu3: 1024KB 64b/line 16-way L2 cache cpu3: CRC32,ASID16 efi0 at mainbus0: UEFI 2.8 efi0: Das U-Boot rev 0x20210100 apm0 at mainbus0 "system" at mainbus0 not configured "axi" at mainbus0 not configured simplebus0 at mainbus0: "soc" bcmclock0 at simplebus0 bcmmbox0 at simplebus0 bcmgpio0 at simplebus0 bcmaux0 at simplebus0 ampintc0 at simplebus0 nirq 256, ncpu 4 ipi: 0, 1: "interrupt-controller" bcmtmon0 at simplebus0 bcmdmac0 at simplebus0: DMA0 DMA2 DMA4 DMA5 DMA6 DMA7 DMA8 DMA9 "timer" at simplebus0 not configured pluart0 at simplebus0: console "local_intc" at simplebus0 not configured bcmdog0 at simplebus0 bcmirng0 at simplebus0 "firmware" at simplebus0 not configured "power" at simplebus0 not configured "mailbox" at simplebus0 not configured sdhc0 at simplebus0 sdhc0: SDHC 3.0, 250 MHz base clock sdmmc0 at sdhc0: 4-bit, sd high-speed, mmc high-speed "gpiomem" at simplebus0 not configured "fb" at simplebus0 not configured "vcsm" at simplebus0 not configured "clocks" at mainbus0 not configured "phy" at mainbus0 not configured "clk-27M" at mainbus0 not configured "clk-108M" at mainbus0 not configured simplebus1 at mainbus0: "emmc2bus" sdhc1 at simplebus1 sdhc1: SDHC 3.0, 100 MHz base clock sdmmc1 at sdhc1: 8-bit, sd high-speed, mmc high-speed, ddr52, dma "arm-pmu" at mainbus0 not configured agtimer0 at mainbus0: 54000 kHz simplebus2 at mainbus0: "scb" bcmpcie0 at simplebus2 pci0 at bcmpcie0 ppb0 at pci0 dev 0 function 0 "Broadcom BCM2711" rev 0x10 pci1 at ppb0 bus 1 xhci0 at pci1 dev 0 function 0 "VIA VL805 xHCI" rev 0x01: intx, xHCI 1.0 usb0 at xhci0: USB revision 3.0 uhub0 at usb0 configuration 1 interface 0 "VIA xHCI root hub" rev 3.00/1.00 addr 1 bse0 at simplebus2: address dc:a6:32:e0:50:d3 brgphy0 at bse0 phy 1: BCM54210E 10/100/1000baseT PHY, rev. 2 "dma" at simplebus2 not configured "hevc-decoder" at simplebus2 not configured "rpivid-local-intc" at simplebus2 not configured "h264-decoder" at simplebus2 not configured "vp9-decoder" at simplebus2 not configured gpioleds0 at mainbus0: "led0", "led1" "sd_io_1v8_reg" at mainbus0 not configured "sd_vcc_reg" at mainbus0 not configured "fixedregulator_3v3" at mainbus0 not configured "fixedregulator_5v0" at mainbus0 not configured simplebus3 at mainbus0: "v3dbus" "bootloader" at mainbus0 not configured dt: 451 probes scsibus0 at sdmmc1: 2 targets, initiator 0 sd0 at scsibus0 targ 1 lun 0: removable sd0: 30436MB, 512 bytes/sector, 62333952 sectors uhub1 at uhub0 port 1 configuration 1 interface 0 "VIA Labs USB2.0 Hub" rev 2.10/4.21 addr 2 bwfm0 at sdmmc0 function 1 manufacturer 0x02d0, product 0xa9a6 at sdmmc0 function 2 not configured manufacturer 0x02d0, product 0xa9a6 at sdmmc0 function 3 not configured vscsi0 at root scsibus1 at vscsi0: 256 targets softraid0 at root scsibus2 at softraid0: 256 targets root on sd0a (d1a7e0233ab9545d.a) swap on sd0b dump on sd0b WARNING: CHECK AND RESET THE DATE! gpio0 at bcmgpio0: 58 pins bwfm0: address dc:a6:32:e0:50:d4
Re: Please test: UVM fault unlocking (aka vmobjlock)
On Tue, Nov 30, 2021 at 07:17:14PM +0100, Jan Stary wrote: > > While here, I notice that sysctl.witness.watch is documented > in witness(4), but kern.witness.locktrace is not - is that intended? > It is documented on options(4) option WITNESS_LOCKTRACE Enable witness(4) lock stack trace saving at boot. The feature is disabled by default and has to be enabled by setting the kern.witness.locktrace sysctl(8) variable. Thanks. -- Sebastien Marie
Re: Please test: UVM fault unlocking (aka vmobjlock)
On Nov 30 10:40:25, s...@spacehopper.org wrote: > On 2021/11/29 22:50, Martin Pieuchot wrote: > > On 24/11/21(Wed) 11:16, Martin Pieuchot wrote: > > > Diff below unlock the bottom part of the UVM fault handler. I'm > > > interested in squashing the remaining bugs. Please test with your usual > > > setup & report back. > > > > Thanks to all the testers, here's a new version that includes a bug fix. > > Tests on !x86 architectures are much appreciated! > This is current/amd64 with the patch on a PC (dmesg below); this is a work machine, running chrome and all, with kern.witness.watch=2 kern.witness.locktrace=0 Shortly after boot, dmesg says witness: lock order reversal: 1st 0xfd83bbfd8788 uobjlk (>vmobjlock) 2nd 0x80b74f00 objmm (>mm.lock) lock order ">mm.lock"(rwlock) -> ">vmobjlock"(rwlock) first seen at: #0 rw_enter+0x65 #1 uvm_obj_wire+0x46 #2 shmem_get_pages+0xaf #3 __i915_gem_object_get_pages+0x9d #4 i915_vma_pin_ww+0x49b #5 i915_ggtt_pin+0x61 #6 __intel_context_do_pin_ww+0x2d1 #7 __intel_context_do_pin+0x4b #8 intel_engines_init+0x4a1 #9 intel_gt_init+0x133 #10 i915_gem_init+0xa3 #11 i915_driver_probe+0x75a #12 inteldrm_attachhook+0x45 #13 config_process_deferred_mountroot+0x6b #14 main+0x743 lock order ">vmobjlock"(rwlock) -> ">mm.lock"(rwlock) first seen at: #0 rw_enter+0x65 #1 __i915_gem_object_get_pages+0x30 #2 i915_gem_fault+0x1aa #3 drm_fault+0x156 #4 uvm_fault+0x179 #5 upageflttrap+0x62 #6 usertrap+0x18b #7 recall_trap+0x8 While here, I notice that sysctl.witness.watch is documented in witness(4), but kern.witness.locktrace is not - is that intended? Jan OpenBSD 7.0-current (GENERIC.MP) #0: Tue Nov 30 15:37:35 CET 2021 h...@box.stare.cz:/usr/src/sys/arch/amd64/compile/GENERIC.MP real mem = 17009606656 (16221MB) avail mem = 16347242496 (15589MB) random: good seed from bootblocks mpath0 at root scsibus0 at mpath0: 256 targets mainbus0 at root bios0 at mainbus0: SMBIOS rev. 2.4 @ 0xf0100 (36 entries) bios0: vendor Award Software International, Inc. version "F3" date 03/31/2011 bios0: Gigabyte Technology Co., Ltd. H67MA-USB3-B3 acpi0 at bios0: ACPI 1.0 acpi0: sleep states S0 S3 S4 S5 acpi0: tables DSDT FACP HPET MCFG ASPT SSPT EUDS MATS TAMG APIC SSDT acpi0: wakeup devices PCI0(S5) PEX0(S5) PEX1(S5) PEX2(S5) PEX3(S5) PEX4(S5) PEX5(S5) PEX6(S5) PEX7(S5) HUB0(S5) UAR1(S3) USBE(S3) USE2(S3) AZAL(S5) acpitimer0 at acpi0: 3579545 Hz, 24 bits acpihpet0 at acpi0: 14318179 Hz acpimcfg0 at acpi0 acpimcfg0: addr 0xf400, bus 0-63 acpimadt0 at acpi0 addr 0xfee0: PC-AT compat cpu0 at mainbus0: apid 0 (boot processor) cpu0: Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz, 3392.72 MHz, 06-2a-07 cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN cpu0: 256KB 64b/line 8-way L2 cache cpu0: smt 0, core 0, package 0 mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges cpu0: apic clock running at 99MHz cpu0: mwait min=64, max=64, C-substates=0.2.1.1, IBE cpu1 at mainbus0: apid 2 (application processor) cpu1: Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz, 3392.31 MHz, 06-2a-07 cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN cpu1: 256KB 64b/line 8-way L2 cache cpu1: smt 0, core 1, package 0 cpu2 at mainbus0: apid 4 (application processor) cpu2: Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz, 3392.31 MHz, 06-2a-07 cpu2: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN cpu2: 256KB 64b/line 8-way L2 cache cpu2: smt 0, core 2, package 0 cpu3 at mainbus0: apid 6 (application processor) cpu3: Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz, 3392.31 MHz, 06-2a-07 cpu3: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN cpu3: 256KB 64b/line 8-way L2 cache cpu3: smt 0, core 3, package 0 cpu4 at mainbus0: apid 1 (application processor) cpu4: Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz, 3392.30 MHz, 06-2a-07 cpu4:
Re: Please test: UVM fault unlocking (aka vmobjlock)
On Mon, Nov 29, 2021 at 10:50:32PM +0100, Martin Pieuchot wrote: > On 24/11/21(Wed) 11:16, Martin Pieuchot wrote: > > Diff below unlock the bottom part of the UVM fault handler. I'm > > interested in squashing the remaining bugs. Please test with your usual > > setup & report back. > > Thanks to all the testers, here's a new version that includes a bug fix. Passed amd64 regress with two witness reports. I don't know whether they were there without the diff. witness: lock order reversal: 1st 0xfd87764d68c8 vmmaplk (>lock) 2nd 0xfd873a6516f8 inode (>i_lock) lock order ">i_lock"(rrwlock) -> ">lock"(rwlock) first seen at: #0 rw_enter_read+0x38 #1 uvmfault_lookup+0x92 #2 uvm_fault_check+0x32 #3 uvm_fault+0xfb #4 kpageflttrap+0x12b #5 kerntrap+0x91 #6 alltraps_kern_meltdown+0x7b #7 copyout+0x53 #8 ffs_read+0x1f6 #9 VOP_READ+0x41 #10 vn_rdwr+0xa1 #11 vmcmd_map_readvn+0xa6 #12 exec_process_vmcmds+0x84 #13 sys_execve+0x79f #14 start_init+0x29f #15 proc_trampoline+0x1c lock order ">lock"(rwlock) -> ">i_lock"(rrwlock) first seen at: #0 rw_enter+0x65 #1 rrw_enter+0x56 #2 VOP_LOCK+0x5b #3 vn_lock+0xad #4 uvn_io+0x1e7 #5 uvm_pager_put+0xf7 #6 uvn_flush+0x260 #7 uvm_map_clean+0x21d #8 syscall+0x3a9 #9 Xsyscall+0x128 witness: lock order reversal: 1st 0xfd886a277768 vmmaplk (>lock) 2nd 0x800022c46360 nfsnode (>n_lock) lock order data w2 -> w1 missing lock order ">lock"(rwlock) -> ">n_lock"(rrwlock) first seen at: #0 rw_enter+0x65 #1 rrw_enter+0x56 #2 VOP_LOCK+0x5b #3 vn_lock+0xad #4 vn_rdwr+0x7f #5 vndstrategy+0x2e6 #6 physio+0x227 #7 spec_write+0x95 #8 VOP_WRITE+0x41 #9 vn_write+0xfc #10 dofilewritev+0x14d #11 sys_pwrite+0x5c #12 syscall+0x3a9 #13 Xsyscall+0x128 bluhm
Re: Please test: UVM fault unlocking (aka vmobjlock)
On 2021/11/29 22:50, Martin Pieuchot wrote: > On 24/11/21(Wed) 11:16, Martin Pieuchot wrote: > > Diff below unlock the bottom part of the UVM fault handler. I'm > > interested in squashing the remaining bugs. Please test with your usual > > setup & report back. > > Thanks to all the testers, here's a new version that includes a bug fix. > > Tests on !x86 architectures are much appreciated! Running here on amd64 and arm64 (plus I added WITNESS to kernel config on arm64 which is only for x86 in your diff). No failures so far, though the machine is quiet (mostly just running conserver + NSD) so it's not the best test ever - if somebody has a more active machine that would be good to get tested. i386 will get a good workout shortly (mkr'ing now). I have one lock order reversal showing at boot on arm64: witness: lock order reversal: 1st 0xff800dab18c0 vmmaplk (>lock) 2nd 0xff800da32098 inode (>i_lock) lock order ">i_lock"(rrwlock) -> ">lock"(rwlock) first seen at: #0 rw_enter_read+0x44 #1 uvmfault_lookup+0xd8 #2 uvm_fault_check+0x40 #3 uvm_fault+0xdc #4 do_el1h_sync+0x118 #5 handle_el1h_sync+0x70 #6 uiomove+0x9c #7 ffs_read+0x1cc #8 VOP_READ+0x3c #9 vn_rdwr+0xac #10 vmcmd_map_readvn+0x8c #11 exec_process_vmcmds+0x84 #12 sys_execve+0x610 #13 start_init+0x23c #14 proc_trampoline+0x14 lock order ">lock"(rwlock) -> ">i_lock"(rrwlock) first seen at: #0 rw_enter+0x84 #1 rrw_enter+0x60 #2 VOP_LOCK+0x64 #3 vn_lock+0xa8 #4 uvn_io+0x19c #5 uvm_pager_put+0xf0 #6 uvn_flush+0x274 #7 uvm_map_clean+0x228 #8 svc_handler+0x310 #9 do_el0_sync+0xe0 #10 handle_el0_sync+0x78 OpenBSD 7.0-current (GENERIC.MP) #4: Tue Nov 30 10:01:15 GMT 2021 st...@mala.spacehopper.org:/sys/arch/arm64/compile/GENERIC.MP real mem = 4134567936 (3943MB) avail mem = 3937853440 (3755MB) random: good seed from bootblocks mainbus0 at root: Raspberry Pi 4 Model B Rev 1.2 psci0 at mainbus0: PSCI 1.1, SMCCC 1.2 cpu0 at mainbus0 mpidr 0: ARM Cortex-A72 r0p3 cpu0: 48KB 64b/line 3-way L1 PIPT I-cache, 32KB 64b/line 2-way L1 D-cache cpu0: 1024KB 64b/line 16-way L2 cache cpu0: CRC32,ASID16 cpu1 at mainbus0 mpidr 1: ARM Cortex-A72 r0p3 cpu1: 48KB 64b/line 3-way L1 PIPT I-cache, 32KB 64b/line 2-way L1 D-cache cpu1: 1024KB 64b/line 16-way L2 cache cpu1: CRC32,ASID16 cpu2 at mainbus0 mpidr 2: ARM Cortex-A72 r0p3 cpu2: 48KB 64b/line 3-way L1 PIPT I-cache, 32KB 64b/line 2-way L1 D-cache cpu2: 1024KB 64b/line 16-way L2 cache cpu2: CRC32,ASID16 cpu3 at mainbus0 mpidr 3: ARM Cortex-A72 r0p3 cpu3: 48KB 64b/line 3-way L1 PIPT I-cache, 32KB 64b/line 2-way L1 D-cache cpu3: 1024KB 64b/line 16-way L2 cache cpu3: CRC32,ASID16 efi0 at mainbus0: UEFI 2.7 efi0: https://github.com/pftf/RPi4 rev 0x1 smbios0 at efi0: SMBIOS 3.3.0 smbios0: vendor https://github.com/pftf/RPi4 version "UEFI Firmware v1.31" date 09/09/2021 smbios0: Raspberry Pi Foundation Raspberry Pi 4 Model B apm0 at mainbus0 "system" at mainbus0 not configured "axi" at mainbus0 not configured simplebus0 at mainbus0: "soc" bcmclock0 at simplebus0 bcmmbox0 at simplebus0 bcmgpio0 at simplebus0 bcmaux0 at simplebus0 ampintc0 at simplebus0 nirq 256, ncpu 4 ipi: 0, 1: "interrupt-controller" bcmtmon0 at simplebus0 bcmdmac0 at simplebus0: DMA0 DMA2 DMA4 DMA5 DMA6 DMA7 DMA8 DMA9 "timer" at simplebus0 not configured pluart0 at simplebus0: console com0 at simplebus0: ns16550, no working fifo "local_intc" at simplebus0 not configured bcmdog0 at simplebus0 bcmirng0 at simplebus0 "firmware" at simplebus0 not configured "power" at simplebus0 not configured "mailbox" at simplebus0 not configured sdhc0 at simplebus0 sdhc0: SDHC 3.0, 250 MHz base clock sdmmc0 at sdhc0: 4-bit, sd high-speed, mmc high-speed "gpiomem" at simplebus0 not configured "fb" at simplebus0 not configured "vcsm" at simplebus0 not configured "clocks" at mainbus0 not configured "phy" at mainbus0 not configured "clk-27M" at mainbus0 not configured "clk-108M" at mainbus0 not configured simplebus1 at mainbus0: "emmc2bus" sdhc1 at simplebus1 sdhc1: SDHC 3.0, 100 MHz base clock sdmmc1 at sdhc1: 8-bit, sd high-speed, mmc high-speed, ddr52, dma "arm-pmu" at mainbus0 not configured agtimer0 at mainbus0: 54000 kHz simplebus2 at mainbus0: "scb" bcmpcie0 at simplebus2 pci0 at bcmpcie0 ppb0 at pci0 dev 0 function 0 "Broadcom BCM2711" rev 0x10 pci1 at ppb0 bus 1 xhci0 at pci1 dev 0 function 0 "VIA VL805 xHCI" rev 0x01: intx, xHCI 1.0 usb0 at xhci0: USB revision 3.0 uhub0 at usb0 configuration 1 interface 0 "VIA xHCI root hub" rev 3.00/1.00 addr 1 bse0 at simplebus2: address dc:a6:32:8b:e1:b7 brgphy0 at bse0 phy 1: BCM54210E 10/100/1000baseT PHY, rev. 2 "dma" at simplebus2 not configured "hevc-decoder" at simplebus2 not configured "rpivid-local-intc" at simplebus2 not configured "h264-decoder" at simplebus2 not configured "vp9-decoder" at simplebus2 not configured gpioleds0 at mainbus0: "led0", "led1" "sd_io_1v8_reg" at mainbus0 not configured "sd_vcc_reg" at mainbus0 not configured "fixedregulator_3v3" at mainbus0 not configured
Re: Please test: UVM fault unlocking (aka vmobjlock)
On 24/11/21(Wed) 11:16, Martin Pieuchot wrote:
> Diff below unlock the bottom part of the UVM fault handler. I'm
> interested in squashing the remaining bugs. Please test with your usual
> setup & report back.
Thanks to all the testers, here's a new version that includes a bug fix.
Tests on !x86 architectures are much appreciated!
Thanks a lot,
Martin
diff --git sys/arch/amd64/conf/GENERIC.MP sys/arch/amd64/conf/GENERIC.MP
index bb842f6d96e..e5334c19eac 100644
--- sys/arch/amd64/conf/GENERIC.MP
+++ sys/arch/amd64/conf/GENERIC.MP
@@ -4,6 +4,6 @@ include "arch/amd64/conf/GENERIC"
option MULTIPROCESSOR
#optionMP_LOCKDEBUG
-#optionWITNESS
+option WITNESS
cpu* at mainbus?
diff --git sys/arch/i386/conf/GENERIC.MP sys/arch/i386/conf/GENERIC.MP
index 980a572b8fd..ef7ded61501 100644
--- sys/arch/i386/conf/GENERIC.MP
+++ sys/arch/i386/conf/GENERIC.MP
@@ -7,6 +7,6 @@ include "arch/i386/conf/GENERIC"
option MULTIPROCESSOR # Multiple processor support
#optionMP_LOCKDEBUG
-#optionWITNESS
+option WITNESS
cpu* at mainbus?
diff --git sys/dev/pci/drm/i915/gem/i915_gem_shmem.c
sys/dev/pci/drm/i915/gem/i915_gem_shmem.c
index ce8e2eca141..47b567087e7 100644
--- sys/dev/pci/drm/i915/gem/i915_gem_shmem.c
+++ sys/dev/pci/drm/i915/gem/i915_gem_shmem.c
@@ -268,8 +268,10 @@ shmem_truncate(struct drm_i915_gem_object *obj)
#ifdef __linux__
shmem_truncate_range(file_inode(obj->base.filp), 0, (loff_t)-1);
#else
+ rw_enter(obj->base.uao->vmobjlock, RW_WRITE);
obj->base.uao->pgops->pgo_flush(obj->base.uao, 0, obj->base.size,
PGO_ALLPAGES | PGO_FREE);
+ rw_exit(obj->base.uao->vmobjlock);
#endif
obj->mm.madv = __I915_MADV_PURGED;
obj->mm.pages = ERR_PTR(-EFAULT);
diff --git sys/dev/pci/drm/radeon/radeon_ttm.c
sys/dev/pci/drm/radeon/radeon_ttm.c
index eb879b5c72c..837a9f94298 100644
--- sys/dev/pci/drm/radeon/radeon_ttm.c
+++ sys/dev/pci/drm/radeon/radeon_ttm.c
@@ -1006,6 +1006,8 @@ radeon_ttm_fault(struct uvm_faultinfo *ufi, vaddr_t
vaddr, vm_page_t *pps,
struct radeon_device *rdev;
int r;
+ KASSERT(rw_write_held(ufi->entry->object.uvm_obj->vmobjlock));
+
bo = (struct drm_gem_object *)ufi->entry->object.uvm_obj;
rdev = bo->dev->dev_private;
down_read(>pm.mclk_lock);
diff --git sys/uvm/uvm_aobj.c sys/uvm/uvm_aobj.c
index 20051d95dc1..a5c403ab67d 100644
--- sys/uvm/uvm_aobj.c
+++ sys/uvm/uvm_aobj.c
@@ -184,7 +184,7 @@ const struct uvm_pagerops aobj_pager = {
* deadlock.
*/
static LIST_HEAD(aobjlist, uvm_aobj) uao_list =
LIST_HEAD_INITIALIZER(uao_list);
-static struct mutex uao_list_lock = MUTEX_INITIALIZER(IPL_NONE);
+static struct mutex uao_list_lock = MUTEX_INITIALIZER(IPL_MPFLOOR);
/*
@@ -277,6 +277,7 @@ uao_find_swslot(struct uvm_object *uobj, int pageidx)
* uao_set_swslot: set the swap slot for a page in an aobj.
*
* => setting a slot to zero frees the slot
+ * => object must be locked by caller
* => we return the old slot number, or -1 if we failed to allocate
*memory to record the new slot number
*/
@@ -286,7 +287,7 @@ uao_set_swslot(struct uvm_object *uobj, int pageidx, int
slot)
struct uvm_aobj *aobj = (struct uvm_aobj *)uobj;
int oldslot;
- KERNEL_ASSERT_LOCKED();
+ KASSERT(rw_write_held(uobj->vmobjlock) || uobj->uo_refs == 0);
KASSERT(UVM_OBJ_IS_AOBJ(uobj));
/*
@@ -358,7 +359,9 @@ uao_free(struct uvm_aobj *aobj)
struct uvm_object *uobj = >u_obj;
KASSERT(UVM_OBJ_IS_AOBJ(uobj));
+ KASSERT(rw_write_held(uobj->vmobjlock));
uao_dropswap_range(uobj, 0, 0);
+ rw_exit(uobj->vmobjlock);
if (UAO_USES_SWHASH(aobj)) {
/*
@@ -671,6 +674,7 @@ struct uvm_object *
uao_create(vsize_t size, int flags)
{
static struct uvm_aobj kernel_object_store;
+ static struct rwlock bootstrap_kernel_object_lock;
static int kobj_alloced = 0;
int pages = round_page(size) >> PAGE_SHIFT;
struct uvm_aobj *aobj;
@@ -742,6 +746,11 @@ uao_create(vsize_t size, int flags)
* Initialise UVM object.
*/
uvm_obj_init(>u_obj, _pager, refs);
+ if (flags & UAO_FLAG_KERNOBJ) {
+ /* Use a temporary static lock for kernel_object. */
+ rw_init(_kernel_object_lock, "kobjlk");
+ uvm_obj_setlock(>u_obj, _kernel_object_lock);
+ }
/*
* now that aobj is ready, add it to the global list
@@ -822,20 +831,20 @@ uao_detach(struct uvm_object *uobj)
* involved in is complete), release any swap resources and free
* the page itself.
*/
- uvm_lock_pageq();
- while((pg = RBT_ROOT(uvm_objtree, >memt)) != NULL) {
+ rw_enter(uobj->vmobjlock, RW_WRITE);
+ while ((pg = RBT_ROOT(uvm_objtree, >memt)) != NULL) {
+ pmap_page_protect(pg, PROT_NONE);
Re: Please test: UVM fault unlocking (aka vmobjlock)
Hi Martin, * Martin Pieuchot wrote: > Diff below unlock the bottom part of the UVM fault handler. I'm > interested in squashing the remaining bugs. Please test with your usual > setup & report back. I have the diff running on a Thinkpad T450s and noticed no regessions so far. Will continue running the diff the next days. Upon boot there was a lock order reversal in dmesg: witness: lock order reversal: 1st 0xfd8351602630 uobjlk (>vmobjlock) 2nd 0x80e53300 objmm (>mm.lock) lock order ">mm.lock"(rwlock) -> ">vmobjlock"(rwlock) first seen at: #0 rw_enter+0x65 #1 uvm_obj_wire+0x46 #2 shmem_get_pages+0xaf #3 __i915_gem_object_get_pages+0x9d #4 i915_vma_pin_ww+0x49b #5 i915_ggtt_pin+0x61 #6 intel_execlists_submission_setup+0x3b2 #7 intel_engines_init+0x2ff #8 intel_gt_init+0x133 #9 i915_gem_init+0xa3 #10 i915_driver_probe+0x75a #11 inteldrm_attachhook+0x45 #12 config_process_deferred_mountroot+0x6b #13 main+0x743 lock order ">vmobjlock"(rwlock) -> ">mm.lock"(rwlock) first seen at: #0 rw_enter+0x65 #1 __i915_gem_object_get_pages+0x30 #2 i915_gem_fault+0x1aa #3 drm_fault+0x156 #4 uvm_fault+0x179 #5 upageflttrap+0x62 #6 usertrap+0x18b #7 recall_trap+0x8 Cheers Matthias OpenBSD 7.0-current (GENERIC.MP) #4: Wed Nov 24 15:33:37 CET 2021 x...@sigma.xosc.net:/usr/src/sys/arch/amd64/compile/GENERIC.MP real mem = 12765257728 (12173MB) avail mem = 12262477824 (11694MB) random: good seed from bootblocks mpath0 at root scsibus0 at mpath0: 256 targets mainbus0 at root bios0 at mainbus0: SMBIOS rev. 2.7 @ 0x9cbfd000 (65 entries) bios0: vendor LENOVO version "JBET73WW (1.37 )" date 08/14/2019 bios0: LENOVO 20BX0049GE acpi0 at bios0: ACPI 5.0 acpi0: sleep states S0 S3 S4 S5 acpi0: tables DSDT FACP SLIC ASF! HPET ECDT APIC MCFG SSDT SSDT SSDT SSDT SSDT SSDT SSDT SSDT SSDT PCCT SSDT TCPA SSDT UEFI MSDM BATB FPDT UEFI DMAR acpi0: wakeup devices LID_(S4) SLPB(S3) IGBE(S4) EXP2(S4) XHCI(S3) EHC1(S3) acpitimer0 at acpi0: 3579545 Hz, 24 bits acpihpet0 at acpi0: 14318179 Hz acpiec0 at acpi0 acpimadt0 at acpi0 addr 0xfee0: PC-AT compat cpu0 at mainbus0: apid 0 (boot processor) cpu0: Intel(R) Core(TM) i5-5200U CPU @ 2.20GHz, 2095.41 MHz, 06-3d-04 cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,RDSEED,ADX,SMAP,PT,SRBDS_CTRL,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN cpu0: 256KB 64b/line 8-way L2 cache cpu0: smt 0, core 0, package 0 mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges cpu0: apic clock running at 99MHz cpu0: mwait min=64, max=64, C-substates=0.2.1.2.4.1.1.1, IBE cpu1 at mainbus0: apid 1 (application processor) cpu1: Intel(R) Core(TM) i5-5200U CPU @ 2.20GHz, 2095.16 MHz, 06-3d-04 cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,RDSEED,ADX,SMAP,PT,SRBDS_CTRL,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN cpu1: 256KB 64b/line 8-way L2 cache cpu1: smt 1, core 0, package 0 cpu2 at mainbus0: apid 2 (application processor) cpu2: Intel(R) Core(TM) i5-5200U CPU @ 2.20GHz, 2095.17 MHz, 06-3d-04 cpu2: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,RDSEED,ADX,SMAP,PT,SRBDS_CTRL,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN cpu2: 256KB 64b/line 8-way L2 cache cpu2: smt 0, core 1, package 0 cpu3 at mainbus0: apid 3 (application processor) cpu3: Intel(R) Core(TM) i5-5200U CPU @ 2.20GHz, 2095.16 MHz, 06-3d-04 cpu3: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,RDSEED,ADX,SMAP,PT,SRBDS_CTRL,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN cpu3: 256KB 64b/line 8-way L2 cache cpu3: smt 1, core 1, package 0 ioapic0 at mainbus0: apid 2 pa 0xfec0, version 20, 40 pins acpimcfg0 at acpi0 acpimcfg0: addr 0xf800, bus
Please test: UVM fault unlocking (aka vmobjlock)
Diff below unlock the bottom part of the UVM fault handler. I'm
interested in squashing the remaining bugs. Please test with your usual
setup & report back.
Thanks,
Martin
diff --git sys/arch/amd64/conf/GENERIC.MP sys/arch/amd64/conf/GENERIC.MP
index bb842f6d96e..e5334c19eac 100644
--- sys/arch/amd64/conf/GENERIC.MP
+++ sys/arch/amd64/conf/GENERIC.MP
@@ -4,6 +4,6 @@ include "arch/amd64/conf/GENERIC"
option MULTIPROCESSOR
#optionMP_LOCKDEBUG
-#optionWITNESS
+option WITNESS
cpu* at mainbus?
diff --git sys/arch/i386/conf/GENERIC.MP sys/arch/i386/conf/GENERIC.MP
index 980a572b8fd..ef7ded61501 100644
--- sys/arch/i386/conf/GENERIC.MP
+++ sys/arch/i386/conf/GENERIC.MP
@@ -7,6 +7,6 @@ include "arch/i386/conf/GENERIC"
option MULTIPROCESSOR # Multiple processor support
#optionMP_LOCKDEBUG
-#optionWITNESS
+option WITNESS
cpu* at mainbus?
diff --git sys/dev/pci/drm/i915/gem/i915_gem_shmem.c
sys/dev/pci/drm/i915/gem/i915_gem_shmem.c
index ce8e2eca141..47b567087e7 100644
--- sys/dev/pci/drm/i915/gem/i915_gem_shmem.c
+++ sys/dev/pci/drm/i915/gem/i915_gem_shmem.c
@@ -268,8 +268,10 @@ shmem_truncate(struct drm_i915_gem_object *obj)
#ifdef __linux__
shmem_truncate_range(file_inode(obj->base.filp), 0, (loff_t)-1);
#else
+ rw_enter(obj->base.uao->vmobjlock, RW_WRITE);
obj->base.uao->pgops->pgo_flush(obj->base.uao, 0, obj->base.size,
PGO_ALLPAGES | PGO_FREE);
+ rw_exit(obj->base.uao->vmobjlock);
#endif
obj->mm.madv = __I915_MADV_PURGED;
obj->mm.pages = ERR_PTR(-EFAULT);
diff --git sys/dev/pci/drm/radeon/radeon_ttm.c
sys/dev/pci/drm/radeon/radeon_ttm.c
index eb879b5c72c..837a9f94298 100644
--- sys/dev/pci/drm/radeon/radeon_ttm.c
+++ sys/dev/pci/drm/radeon/radeon_ttm.c
@@ -1006,6 +1006,8 @@ radeon_ttm_fault(struct uvm_faultinfo *ufi, vaddr_t
vaddr, vm_page_t *pps,
struct radeon_device *rdev;
int r;
+ KASSERT(rw_write_held(ufi->entry->object.uvm_obj->vmobjlock));
+
bo = (struct drm_gem_object *)ufi->entry->object.uvm_obj;
rdev = bo->dev->dev_private;
down_read(>pm.mclk_lock);
diff --git sys/uvm/uvm_aobj.c sys/uvm/uvm_aobj.c
index 20051d95dc1..127218c4c40 100644
--- sys/uvm/uvm_aobj.c
+++ sys/uvm/uvm_aobj.c
@@ -31,7 +31,7 @@
/*
* uvm_aobj.c: anonymous memory uvm_object pager
*
- * author: Chuck Silvers
+* author: Chuck Silvers
* started: Jan-1998
*
* - design mostly from Chuck Cranor
@@ -184,7 +184,7 @@ const struct uvm_pagerops aobj_pager = {
* deadlock.
*/
static LIST_HEAD(aobjlist, uvm_aobj) uao_list =
LIST_HEAD_INITIALIZER(uao_list);
-static struct mutex uao_list_lock = MUTEX_INITIALIZER(IPL_NONE);
+static struct mutex uao_list_lock = MUTEX_INITIALIZER(IPL_MPFLOOR);
/*
@@ -277,6 +277,7 @@ uao_find_swslot(struct uvm_object *uobj, int pageidx)
* uao_set_swslot: set the swap slot for a page in an aobj.
*
* => setting a slot to zero frees the slot
+ * => object must be locked by caller
* => we return the old slot number, or -1 if we failed to allocate
*memory to record the new slot number
*/
@@ -286,7 +287,7 @@ uao_set_swslot(struct uvm_object *uobj, int pageidx, int
slot)
struct uvm_aobj *aobj = (struct uvm_aobj *)uobj;
int oldslot;
- KERNEL_ASSERT_LOCKED();
+ KASSERT(rw_write_held(uobj->vmobjlock) || uobj->uo_refs == 0);
KASSERT(UVM_OBJ_IS_AOBJ(uobj));
/*
@@ -358,7 +359,9 @@ uao_free(struct uvm_aobj *aobj)
struct uvm_object *uobj = >u_obj;
KASSERT(UVM_OBJ_IS_AOBJ(uobj));
+ KASSERT(rw_write_held(uobj->vmobjlock));
uao_dropswap_range(uobj, 0, 0);
+ rw_exit(uobj->vmobjlock);
if (UAO_USES_SWHASH(aobj)) {
/*
@@ -671,6 +674,7 @@ struct uvm_object *
uao_create(vsize_t size, int flags)
{
static struct uvm_aobj kernel_object_store;
+ static struct rwlock bootstrap_kernel_object_lock;
static int kobj_alloced = 0;
int pages = round_page(size) >> PAGE_SHIFT;
struct uvm_aobj *aobj;
@@ -742,6 +746,11 @@ uao_create(vsize_t size, int flags)
* Initialise UVM object.
*/
uvm_obj_init(>u_obj, _pager, refs);
+ if (flags & UAO_FLAG_KERNOBJ) {
+ /* Use a temporary static lock for kernel_object. */
+ rw_init(_kernel_object_lock, "kobjlk");
+ uvm_obj_setlock(>u_obj, _kernel_object_lock);
+ }
/*
* now that aobj is ready, add it to the global list
@@ -822,20 +831,20 @@ uao_detach(struct uvm_object *uobj)
* involved in is complete), release any swap resources and free
* the page itself.
*/
- uvm_lock_pageq();
- while((pg = RBT_ROOT(uvm_objtree, >memt)) != NULL) {
+ rw_enter(uobj->vmobjlock, RW_WRITE);
+ while ((pg = RBT_ROOT(uvm_objtree, >memt)) != NULL) {
+ pmap_page_protect(pg, PROT_NONE);
