Re: Prevent off-by-one accounting hang in out-of-swap situations

2023-10-26 Thread Martin Pieuchot
On 26/10/23(Thu) 07:06, Miod Vallat wrote: > > I wonder if the diff below makes a difference. It's hard to debug and it > > might be worth adding a counter for bad swap slots. > > It did not help (but your diff is probably correct). In that case I'd like to put both diffs in, are you ok with tha

Re: Prevent off-by-one accounting hang in out-of-swap situations

2023-10-22 Thread Martin Pieuchot
On 22/10/23(Sun) 20:29, Miod Vallat wrote: > > On 21/10/23(Sat) 14:28, Miod Vallat wrote: > > > > Stuart, Miod, I wonder if this also help for the off-by-one issue you > > > > are seeing. It might not. > > > > > > It makes the aforementioned issue disappear on the affected machine. > > > > Thank

Re: Prevent off-by-one accounting hang in out-of-swap situations

2023-10-21 Thread Martin Pieuchot
On 21/10/23(Sat) 14:28, Miod Vallat wrote: > > Stuart, Miod, I wonder if this also help for the off-by-one issue you > > are seeing. It might not. > > It makes the aforementioned issue disappear on the affected machine. Thanks at lot for testing! > > Comments, ok? > > > diff --git sys/uvm/uvm_

Re: bt(5), btrace(8): execute END probe and print maps after exit() statement

2023-10-21 Thread Martin Pieuchot
On 18/10/23(Wed) 12:56, Scott Cheloha wrote: > Hi, > > A bt(5) exit() statement causes the btrace(8) interpreter to exit(3) > immediately. > > A BPFtrace exit() statement is more nuanced: the END probe is executed > and the contents of all maps are printed before the interpreter exits. > > This

Prevent off-by-one accounting hang in out-of-swap situations

2023-10-17 Thread Martin Pieuchot
Diff below makes out-of-swap checks more robust. When a system is low on swap, two variables are used to adapt the behavior of the page daemon and fault handler: `swpginuse' indicates how much swap space is being currently used `swpgonly' indicates how much swap space stores cont

small fix in uvmpd_scan_inactive()

2023-10-17 Thread Martin Pieuchot
Diff below merges two equivalent if blocks. No functional change, ok? Index: uvm/uvm_pdaemon.c === RCS file: /cvs/src/sys/uvm/uvm_pdaemon.c,v retrieving revision 1.107 diff -u -p -r1.107 uvm_pdaemon.c --- uvm/uvm_pdaemon.c 16 Oct

Re: dt(4), hardclock(9): move interval, profile providers to dedicated callback

2023-09-18 Thread Martin Pieuchot
mitate what other kernel APIs do. Look at task_add(9) and timeout_add(9). Call the functions add/del() to match existing APIs, then we can add a clockintr_del_barrier() if needed. Do not introduce functions before we need them. I hope we won't need it. > On Mon, Sep 04, 2023 at 01:39:25PM +01

Re: Use counters_read(9) from ddb(4)

2023-09-15 Thread Martin Pieuchot
On 11/09/23(Mon) 21:05, Martin Pieuchot wrote: > On 06/09/23(Wed) 23:13, Alexander Bluhm wrote: > > On Wed, Sep 06, 2023 at 12:23:33PM -0500, Scott Cheloha wrote: > > > On Wed, Sep 06, 2023 at 01:04:19PM +0100, Martin Pieuchot wrote: > > > > Debugging OOM is hard.

Re: Use counters_read(9) from ddb(4)

2023-09-11 Thread Martin Pieuchot
On 06/09/23(Wed) 23:13, Alexander Bluhm wrote: > On Wed, Sep 06, 2023 at 12:23:33PM -0500, Scott Cheloha wrote: > > On Wed, Sep 06, 2023 at 01:04:19PM +0100, Martin Pieuchot wrote: > > > Debugging OOM is hard. UVM uses per-CPU counters and sadly > > > counters_read(

Use counters_read(9) from ddb(4)

2023-09-06 Thread Martin Pieuchot
Debugging OOM is hard. UVM uses per-CPU counters and sadly counters_read(9) needs to allocate memory. This is not acceptable in ddb(4). As a result I cannot see the content of UVM counters in OOM situations. Diff below introduces a *_static() variant of counters_read(9) that takes a secondary b

Re: dt(4), hardclock(9): move interval, profile providers to dedicated callback

2023-09-04 Thread Martin Pieuchot
On 25/08/23(Fri) 21:00, Scott Cheloha wrote: > On Thu, Aug 24, 2023 at 07:21:29PM +0200, Martin Pieuchot wrote: > > [...] > > The only behavior that needs to be preserved is the output of dumping > > stacks. That means DT_FA_PROFILE and DT_FA_STATIC certainly needs to >

Re: dt(4), hardclock(9): move interval, profile providers to dedicated callback

2023-09-04 Thread Martin Pieuchot
On 25/08/23(Fri) 21:00, Scott Cheloha wrote: > On Thu, Aug 24, 2023 at 07:21:29PM +0200, Martin Pieuchot wrote: > > On 23/08/23(Wed) 18:52, Scott Cheloha wrote: > > > This is the next patch in the clock interrupt reorganization series. > > > > Thanks for your dif

Re: anon & pmap_page_protect

2023-09-01 Thread Martin Pieuchot
On 12/08/23(Sat) 10:43, Martin Pieuchot wrote: > Since UVM has been imported, we zap mappings associated to anon pages > before deactivating or freeing them. Sadly, with the introduction of > locking for amaps & anons, I added new code paths that do not respect > this behavior.

Re: dt(4), hardclock(9): move interval, profile providers to dedicated callback

2023-08-24 Thread Martin Pieuchot
On 23/08/23(Wed) 18:52, Scott Cheloha wrote: > This is the next patch in the clock interrupt reorganization series. Thanks for your diff. I'm sorry but it is really hard for me to help review this diff because there is still no man page for this API+subsystem. Can we start with that please? > T

Re: smr_grace_wait(): Skip halted CPUs

2023-08-12 Thread Martin Pieuchot
On 12/08/23(Sat) 11:48, Visa Hankala wrote: > On Sat, Aug 12, 2023 at 01:29:10PM +0200, Martin Pieuchot wrote: > > On 12/08/23(Sat) 10:57, Visa Hankala wrote: > > > On Fri, Aug 11, 2023 at 09:52:15PM +0200, Martin Pieuchot wrote: > > > > When stopping a machi

Re: smr_grace_wait(): Skip halted CPUs

2023-08-12 Thread Martin Pieuchot
On 12/08/23(Sat) 10:57, Visa Hankala wrote: > On Fri, Aug 11, 2023 at 09:52:15PM +0200, Martin Pieuchot wrote: > > When stopping a machine, with "halt -p" for example, secondary CPUs are > > removed from the scheduler before smr_flush() is called. So there's no >

anon & pmap_page_protect

2023-08-12 Thread Martin Pieuchot
Since UVM has been imported, we zap mappings associated to anon pages before deactivating or freeing them. Sadly, with the introduction of locking for amaps & anons, I added new code paths that do not respect this behavior. The diff below restores it by moving the call to pmap_page_protect() insid

smr_grace_wait(): Skip halted CPUs

2023-08-11 Thread Martin Pieuchot
When stopping a machine, with "halt -p" for example, secondary CPUs are removed from the scheduler before smr_flush() is called. So there's no need for the SMR thread to peg itself to such CPUs. This currently isn't a problem because we use per-CPU runqueues but it doesn't work with a global one.

Re: uvm_pagelookup(): moar sanity checks

2023-08-11 Thread Martin Pieuchot
On 11/08/23(Fri) 20:41, Mark Kettenis wrote: > > Date: Fri, 11 Aug 2023 20:12:19 +0200 > > From: Martin Pieuchot > > > > Here's a simple diff to add some more sanity checks in uvm_pagelookup(). > > > > Nothing fancy, it helps documenting the flags

uvm_pagelookup(): moar sanity checks

2023-08-11 Thread Martin Pieuchot
Here's a simple diff to add some more sanity checks in uvm_pagelookup(). Nothing fancy, it helps documenting the flags and reduce the difference with NetBSD. This is part of my on-going work on UVM. ok? Index: uvm/uvm_page.c === RC

Re: hardclock(9), roundrobin: make roundrobin() an independent clock interrupt

2023-08-10 Thread Martin Pieuchot
On 10/08/23(Thu) 12:18, Scott Cheloha wrote: > On Thu, Aug 10, 2023 at 01:05:27PM +0200, Martin Pieuchot wrote: > [...] > > Can we get rid of `hardclock_period' and use a variable set to 100ms? > > This should be tested on alpha which has a hz of 1024 but I'd argu

Re: hardclock(9), roundrobin: make roundrobin() an independent clock interrupt

2023-08-10 Thread Martin Pieuchot
On 05/08/23(Sat) 17:17, Scott Cheloha wrote: > This is the next piece of the clock interrupt reorganization patch > series. The round robin logic is here to make sure process doesn't hog a CPU. The period to tell a process it should yield doesn't have to be tied to the hardclock period. We want t

Re: [v2]: uvm_meter, schedcpu: make uvm_meter() an independent timeout

2023-08-03 Thread Martin Pieuchot
On 02/08/23(Wed) 18:27, Claudio Jeker wrote: > On Wed, Aug 02, 2023 at 10:15:20AM -0500, Scott Cheloha wrote: > > Now that the proc0 wakeup(9) is gone we can retry the other part of > > the uvm_meter() patch. > > > > uvm_meter() is meant to run every 5 seconds, but for historical > > reasons it is

Re: uvm_loadav: don't recompute schedstate_percpu.spc_nrun

2023-08-03 Thread Martin Pieuchot
On 02/08/23(Wed) 14:22, Claudio Jeker wrote: > On Mon, Jul 31, 2023 at 10:21:11AM -0500, Scott Cheloha wrote: > > On Fri, Jul 28, 2023 at 07:36:41PM -0500, Scott Cheloha wrote: > > > claudio@ notes that uvm_loadav() pointlessly walks the allproc list to > > > recompute schedstate_percpu.spn_nrun fo

Re: Make USB WiFi drivers more robust

2023-07-24 Thread Martin Pieuchot
On 24/07/23(Mon) 12:07, Mark Kettenis wrote: > Hi All, > > I recently committed a change to the xhci(4) driver that fixed an > issue with suspending a machine while it has USB devices plugged in. > Unfortunately this diff had some unintended side effects. After > looking at the way the USB stack

Re: patch: atfork unlock

2023-05-21 Thread Martin Pieuchot
On 07/12/22(Wed) 22:17, Joel Knight wrote: > Hi. As first mentioned on misc[1], I've identified a deadlock in libc > when a process forks, the children are multi-threaded, and they set > one or more atfork callbacks. The diff below causes ATFORK_UNLOCK() to > release the lock even when the process

Re: Get rid of UVM_VNODE_CANPERSIST

2022-11-29 Thread Martin Pieuchot
On 28/11/22(Mon) 15:04, Mark Kettenis wrote: > > Date: Wed, 23 Nov 2022 17:33:26 +0100 > > From: Martin Pieuchot > > > > On 23/11/22(Wed) 16:34, Mark Kettenis wrote: > > > > Date: Wed, 23 Nov 2022 10:52:32 +0100 > > > > From: Martin Pieucho

Re: Get rid of UVM_VNODE_CANPERSIST

2022-11-23 Thread Martin Pieuchot
On 23/11/22(Wed) 16:34, Mark Kettenis wrote: > > Date: Wed, 23 Nov 2022 10:52:32 +0100 > > From: Martin Pieuchot > > > > On 22/11/22(Tue) 23:40, Mark Kettenis wrote: > > > > Date: Tue, 22 Nov 2022 17:47:44 + > > > > From: Miod Vallat > >

Re: Get rid of UVM_VNODE_CANPERSIST

2022-11-23 Thread Martin Pieuchot
On 22/11/22(Tue) 23:40, Mark Kettenis wrote: > > Date: Tue, 22 Nov 2022 17:47:44 + > > From: Miod Vallat > > > > > Here is a diff. Maybe bluhm@ can try this on the macppc machine that > > > triggered the original "vref used where vget required" problem? > > > > On a similar machine it panic

Re: Get rid of UVM_VNODE_CANPERSIST

2022-11-22 Thread Martin Pieuchot
On 18/11/22(Fri) 21:33, Mark Kettenis wrote: > > Date: Thu, 17 Nov 2022 20:23:37 +0100 > > From: Mark Kettenis > > > > > From: Jeremie Courreges-Anglas > > > Date: Thu, 17 Nov 2022 18:00:21 +0100 > > > > > > On Tue, Nov 15 2022, Marti

Get rid of UVM_VNODE_CANPERSIST

2022-11-15 Thread Martin Pieuchot
UVM vnode objects include a reference count to keep track of the number of processes that have the corresponding pages mapped in their VM space. When the last process referencing a given library or executable dies, the reaper will munmap this object on its behalf. When this happens it doesn't fre

btrace: string comparison in filters

2022-11-11 Thread Martin Pieuchot
Diff below adds support for the common following idiom: syscall:open:entry /comm == "ksh"/ { ... } String comparison is tricky as it can be combined with any other expression in filters, like: syscall:mmap:entry /comm == "cc" && pid != 4589/ { ... } I don't have the energy to ch

Re: push kernel lock inside ifioctl_get()

2022-11-08 Thread Martin Pieuchot
On 08/11/22(Tue) 15:28, Klemens Nanni wrote: > After this mechanical move, I can unlock the individual SIOCG* in there. I'd suggest grabbing the KERNEL_LOCK() after NET_LOCK_SHARED(). Otherwise you might spin for the first one then release it when going to sleep. > OK? > > Index: if.c >

Mark sched_yield(2) as NOLOCK

2022-11-08 Thread Martin Pieuchot
Now that mmap/munmap/mprotect(2) are no longer creating contention it is possible to see that sched_yield(2) is one of the syscalls waiting for the KERNEL_LOCK() to be released. However this is no longer necessary. Traversing `ps_threads' require either the KERNEL_LOCK() or the SCHED_LOCK() and w

Re: xenstore.c: return error number

2022-11-08 Thread Martin Pieuchot
On 01/11/22(Tue) 15:26, Masato Asou wrote: > Hi, > > Return error number instead of call panic(). Makes sense to me. Do you know how this error can occur? Is is a logic error or are we trusting values produced by a third party? > comment, ok? > -- > ASOU Masato > > diff --git a/sys/dev/pv/xen

Re: Please test: unlock mprotect/mmap/munmap

2022-11-08 Thread Martin Pieuchot
On 08/11/22(Tue) 11:12, Mark Kettenis wrote: > > Date: Tue, 8 Nov 2022 10:32:14 +0100 > > From: Christian Weisgerber > > > > Martin Pieuchot: > > > > > These 3 syscalls should now be ready to run w/o KERNEL_LOCK(). This > > > will reduce c

Re: push kernel lock down in ifioctl()

2022-11-07 Thread Martin Pieuchot
On 07/11/22(Mon) 15:16, Klemens Nanni wrote: > Not all interface ioctls need the kernel lock, but they all grab it. > > Here's a mechanical diff splitting the single lock/unlock around > ifioctl() into individual lock/unlock dances inside ifioctl(). > > From there we can unlock individual ioctls

Please test: unlock mprotect/mmap/munmap

2022-11-06 Thread Martin Pieuchot
These 3 syscalls should now be ready to run w/o KERNEL_LOCK(). This will reduce contention a lot. I'd be happy to hear from test reports on many architectures and possible workloads. Do not forget to run "make syscalls" before building the kernel. Index: syscalls.master

Re: Towards unlocking mmap(2) & munmap(2)

2022-10-30 Thread Martin Pieuchot
On 30/10/22(Sun) 12:45, Klemens Nanni wrote: > On Sun, Oct 30, 2022 at 12:40:02PM +, Klemens Nanni wrote: > > regress on i386/GENERIC.MP+WITNESS with this diff shows > > Another one; This machine has three read-only NFS mounts, but none of > them are used during builds or regress. It's the s

Re: Towards unlocking mmap(2) & munmap(2)

2022-10-30 Thread Martin Pieuchot
On 30/10/22(Sun) 12:40, Klemens Nanni wrote: > On Fri, Oct 28, 2022 at 11:08:55AM +0200, Martin Pieuchot wrote: > > On 20/10/22(Thu) 16:17, Martin Pieuchot wrote: > > > On 11/09/22(Sun) 12:26, Martin Pieuchot wrote: > > > > Diff below adds a minimalist set of ass

Re: Towards unlocking mmap(2) & munmap(2)

2022-10-28 Thread Martin Pieuchot
On 20/10/22(Thu) 16:17, Martin Pieuchot wrote: > On 11/09/22(Sun) 12:26, Martin Pieuchot wrote: > > Diff below adds a minimalist set of assertions to ensure proper locks > > are held in uvm_mapanon() and uvm_unmap_remove() which are the guts of > > mmap(2) for anons and munma

Re: Towards unlocking mmap(2) & munmap(2)

2022-10-20 Thread Martin Pieuchot
On 11/09/22(Sun) 12:26, Martin Pieuchot wrote: > Diff below adds a minimalist set of assertions to ensure proper locks > are held in uvm_mapanon() and uvm_unmap_remove() which are the guts of > mmap(2) for anons and munmap(2). > > Please test it with WITNESS enabled and report bac

Re: Towards unlocking mmap(2) & munmap(2)

2022-09-14 Thread Martin Pieuchot
On 14/09/22(Wed) 15:47, Klemens Nanni wrote: > On 14.09.22 18:55, Mike Larkin wrote: > > On Sun, Sep 11, 2022 at 12:26:31PM +0200, Martin Pieuchot wrote: > > > Diff below adds a minimalist set of assertions to ensure proper locks > > > are held in uvm_mapanon() and uvm_u

Towards unlocking mmap(2) & munmap(2)

2022-09-11 Thread Martin Pieuchot
Diff below adds a minimalist set of assertions to ensure proper locks are held in uvm_mapanon() and uvm_unmap_remove() which are the guts of mmap(2) for anons and munmap(2). Please test it with WITNESS enabled and report back. Index: uvm/uvm_addr.c

uvm_vnode locking & documentation

2022-09-10 Thread Martin Pieuchot
Previous fix from gnezdo@ pointed out that `u_flags' accesses should be serialized by `vmobjlock'. Diff below documents this and fix the remaining places where the lock isn't yet taken. One exception still remains, the first loop of uvm_vnp_sync(). This cannot be fixed right now due to possible

Re: Unmap page in uvm_anon_release()

2022-09-10 Thread Martin Pieuchot
On 10/09/22(Sat) 15:12, Mark Kettenis wrote: > > Date: Sat, 10 Sep 2022 14:18:02 +0200 > > From: Martin Pieuchot > > > > Diff below fixes a bug exposed when swapping on arm64. When an anon is > > released make sure the all the pmap references to the related pa

Unmap page in uvm_anon_release()

2022-09-10 Thread Martin Pieuchot
Diff below fixes a bug exposed when swapping on arm64. When an anon is released make sure the all the pmap references to the related page are removed. We could move the pmap_page_protect(pg, PROT_NONE) inside uvm_pagefree() to avoid future issue but that's for a later refactoring. With this diff

Re: ps(1): add -d (descendancy) option to display parent/child process relationships

2022-09-01 Thread Martin Pieuchot
On 01/09/22(Thu) 03:37, Job Snijders wrote: > Dear all, > > Some ps(1) implementations have an '-d' ('descendancy') option. Through > ASCII art parent/child process relationships are grouped and displayed. > Here is an example: > > $ ps ad -O ppid,user > PID PPID USER TT STAT

Re: pdaemon locking tweak

2022-08-30 Thread Martin Pieuchot
On 30/08/22(Tue) 15:28, Jonathan Gray wrote: > On Mon, Aug 29, 2022 at 01:46:20PM +0200, Martin Pieuchot wrote: > > Diff below refactors the pdaemon's locking by introducing a new *trylock() > > function for a given page. This is shamelessly stolen from NetBSD. > > >

uvmpd_dropswap()

2022-08-29 Thread Martin Pieuchot
Small refactoring to introduce uvmpd_dropswap(). This will make an upcoming rewrite of the pdaemon smaller & easier to review :o) ok? Index: uvm/uvm_pdaemon.c === RCS file: /cvs/src/sys/uvm/uvm_pdaemon.c,v retrieving revision 1.102

pdaemon locking tweak

2022-08-29 Thread Martin Pieuchot
Diff below refactors the pdaemon's locking by introducing a new *trylock() function for a given page. This is shamelessly stolen from NetBSD. This is part of my ongoing effort to untangle the locks used by the page daemon. ok? Index: uvm//uvm_pdaemon.c ==

Simplify locking code in pdaemon

2022-08-18 Thread Martin Pieuchot
Use a "slock" variable as done in multiple places to simplify the code. The locking stay the same. This is just a first step to simplify this mess. Also get rid of the return value of the function, it is never checked. ok? Index: uvm/uvm_pdaemon.c ===

Fix a race in uvm_pseg_release()

2022-08-18 Thread Martin Pieuchot
The lock must be grabbed before iterating on the global array, ok? Index: uvm/uvm_pager.c === RCS file: /cvs/src/sys/uvm/uvm_pager.c,v retrieving revision 1.88 diff -u -p -r1.88 uvm_pager.c --- uvm/uvm_pager.c 15 Aug 2022 03:21:04

Re: uvm_swap: introduce uvm_swap_data_lock

2022-08-17 Thread Martin Pieuchot
On 16/01/22(Sun) 15:35, Martin Pieuchot wrote: > On 30/12/21(Thu) 23:38, Theo Buehler wrote: > > The diff below does two things: it adds a uvm_swap_data_lock mutex and > > trades it for the KERNEL_LOCK in uvm_swapisfull() and uvm_swap_markbad() > > Why is it enough? Whi

Re: patch: change swblk_t type and use it in blist

2022-08-05 Thread Martin Pieuchot
On 05/08/22(Fri) 18:10, Sebastien Marie wrote: > Hi, > > When initially ported blist from DragonFlyBSD, we used custom type bsblk_t > and > bsbmp_t instead of the one used by DragonFlyBSD (swblk_t and u_swblk_t). > > The reason was swblk_t is already defined on OpenBSD, and was incompatible >

Re: Introduce uvm_pagewait()

2022-07-11 Thread Martin Pieuchot
On 28/06/22(Tue) 14:13, Martin Pieuchot wrote: > I'd like to abstract the use of PG_WANTED to start unifying & cleaning > the various cases where a code path is waiting for a busy page. Here's > the first step. > > ok? Any

Faster M operation for the swapper to be great again

2022-06-30 Thread Martin Pieuchot
Diff below uses two tricks to make uvm_pagermapin/out() faster and less likely to fail in OOM situations. These functions are used to map buffers when swapping pages in/out and when faulting on mmaped files. robert@ even measured a 75% improvement when populating pages related to files that aren'

Re: Use SMR instead of SRP list in rtsock.c

2022-06-30 Thread Martin Pieuchot
On 30/06/22(Thu) 11:56, Claudio Jeker wrote: > On Thu, Jun 30, 2022 at 12:34:33PM +0300, Vitaliy Makkoveev wrote: > > On Thu, Jun 30, 2022 at 11:08:48AM +0200, Claudio Jeker wrote: > > > This diff converts the SRP list to a SMR list in rtsock.c > > > SRP is a bit strange with how it works and the S

Re: arp llinfo mutex

2022-06-29 Thread Martin Pieuchot
On 29/06/22(Wed) 19:40, Alexander Bluhm wrote: > Hi, > > To fix the KASSERT(la != NULL) we have to protect the rt_llinfo > with a mutex. The idea is to keep rt_llinfo and RTF_LLINFO consistent. > Also do not put the mutex in the fast path. Losing the RTM_ADD/DELETE race is not a bug. I would no

Simplify aiodone daemon

2022-06-29 Thread Martin Pieuchot
The aiodone daemon accounts for and frees/releases pages they were written to swap. It is only used for asynchronous write. The diff below uses this knowledge to: - Stop suggesting that uvm_swap_get() can be asynchronous. There's an assert for PGO_SYNCIO 3 lines above. - Remove unused suppor

Re: Unlocking pledge(2)

2022-06-28 Thread Martin Pieuchot
On 28/06/22(Tue) 18:17, Jeremie Courreges-Anglas wrote: > > Initially I just wandered in syscall_mi.h and found the locking scheme > super weird, even if technically correct. pledge_syscall() better be > safe to call without the kernel lock so I don't understand why we're > sometimes calling it w

Re: Fix the swapper

2022-06-28 Thread Martin Pieuchot
On 27/06/22(Mon) 15:44, Martin Pieuchot wrote: > Diff below contain 3 parts that can be committed independently. The 3 > of them are necessary to allow the pagedaemon to make progress in OOM > situation and to satisfy all the allocations waiting for pages in > specific ranges

Introduce uvm_pagewait()

2022-06-28 Thread Martin Pieuchot
I'd like to abstract the use of PG_WANTED to start unifying & cleaning the various cases where a code path is waiting for a busy page. Here's the first step. ok? Index: uvm/uvm_amap.c === RCS file: /cvs/src/sys/uvm/uvm_amap.c,v retr

Re: kernel lock in arp

2022-06-27 Thread Martin Pieuchot
On 27/06/22(Mon) 19:11, Alexander Bluhm wrote: > On Mon, Jun 27, 2022 at 11:49:23AM +0200, Alexander Bluhm wrote: > > On Sat, May 21, 2022 at 10:50:28PM +0300, Vitaliy Makkoveev wrote: > > > This diff looks good, except the re-check after kernel lock. It???s > > > supposed `rt??? could became incon

CoW & neighbor pages

2022-06-27 Thread Martin Pieuchot
When faulting a page after COW neighborhood pages are likely to already be entered. So speed up the fault by doing a narrow fault (do not try to map in adjacent pages). This is stolen from NetBSD. ok? Index: uvm/uvm_fault.c === RC

Fix the swapper

2022-06-27 Thread Martin Pieuchot
Diff below contain 3 parts that can be committed independently. The 3 of them are necessary to allow the pagedaemon to make progress in OOM situation and to satisfy all the allocations waiting for pages in specific ranges. * uvm/uvm_pager.c part reserves a second segment for the page daemon. Th

pdaemon: reserve memory for swapping

2022-06-26 Thread Martin Pieuchot
uvm_swap_io() needs to perform up to 4 allocations to write pages to disk. In OOM situation uvm_swap_allocpages() always fail because the kernel doesn't reserve enough pages. Diff below set `uvmexp.reserve_pagedaemon' to the number of pages needed to write a cluster of pages to disk. With this m

Re: set RTF_DONE in sysctl_dumpentry for the routing table

2022-06-08 Thread Martin Pieuchot
On 08/06/22(Wed) 16:13, Claudio Jeker wrote: > Notice while hacking in OpenBGPD. Unlike routing socket messages the > messages from the sysctl interface have RTF_DONE not set. > I think it would make sense to set RTF_DONE also in this case since it > makes reusing code easier. > > All messages sen

Re: Fix clearing of sleep timeouts

2022-06-06 Thread Martin Pieuchot
On 06/06/22(Mon) 06:47, David Gwynne wrote: > On Sun, Jun 05, 2022 at 03:57:39PM +, Visa Hankala wrote: > > On Sun, Jun 05, 2022 at 12:27:32PM +0200, Martin Pieuchot wrote: > > > On 05/06/22(Sun) 05:20, Visa Hankala wrote: > > > > Encountered the following panic:

Re: Fix clearing of sleep timeouts

2022-06-05 Thread Martin Pieuchot
On 05/06/22(Sun) 05:20, Visa Hankala wrote: > Encountered the following panic: > > panic: kernel diagnostic assertion "(p->p_flag & P_TIMEOUT) == 0" failed: > file "/usr/src/sys/kern/kern_synch.c", line 373 > Stopped at db_enter+0x10: popq%rbp > TIDPIDUID PRFLAGS PFL

Re: start unlocking kbind(2)

2022-05-31 Thread Martin Pieuchot
On 18/05/22(Wed) 15:53, Alexander Bluhm wrote: > On Tue, May 17, 2022 at 10:44:54AM +1000, David Gwynne wrote: > > + cookie = SCARG(uap, proc_cookie); > > + if (pr->ps_kbind_addr == pc) { > > + membar_datadep_consumer(); > > + if (pr->ps_kbind_cookie != cookie) > > +

Re: ffs_truncate: Missing uvm_vnp_uncache() w/ softdep

2022-05-24 Thread Martin Pieuchot
On 24/05/22(Tue) 15:24, Mark Kettenis wrote: > > Date: Tue, 24 May 2022 14:28:39 +0200 > > From: Martin Pieuchot > > > > The softdep code path is missing a UVM cache invalidation compared to > > the !softdep one. This is necessary to flush pages of a persis

Please test: rewrite of pdaemon

2022-05-24 Thread Martin Pieuchot
Diff below brings in & adapt most of the changes from NetBSD's r1.37 of uvm_pdaemon.c. My motivation for doing this is to untangle the inner loop of uvmpd_scan_inactive() which will allow us to split the global `pageqlock' mutex in a next step. The idea behind this change is to get rid of the too

ffs_truncate: Missing uvm_vnp_uncache() w/ softdep

2022-05-24 Thread Martin Pieuchot
The softdep code path is missing a UVM cache invalidation compared to the !softdep one. This is necessary to flush pages of a persisting vnode. Since uvm_vnp_setsize() is also called later in this function for the !softdep case move it to not call it twice. ok? Index: ufs/ffs/ffs_inode.c ==

Re: Call uvm_vnp_uncache() before VOP_RENAME()

2022-05-24 Thread Martin Pieuchot
On 17/05/22(Tue) 16:55, Martin Pieuchot wrote: > nfsrv_rename() should behave like dorenameat() and tell UVM to "flush" a > possibly mmap'ed file before calling VOP_RENAME(). > > ok? Anyone? > Index: nfs/nfs_serv.c > =

Re: start unlocking kbind(2)

2022-05-17 Thread Martin Pieuchot
On 17/05/22(Tue) 10:44, David Gwynne wrote: > this narrows the scope of the KERNEL_LOCK in kbind(2) so the syscall > argument checks can be done without the kernel lock. > > care is taken to allow the pc/cookie checks to run without any lock by > being careful with the order of the checks. all mod

Call uvm_vnp_uncache() before VOP_RENAME()

2022-05-17 Thread Martin Pieuchot
nfsrv_rename() should behave like dorenameat() and tell UVM to "flush" a possibly mmap'ed file before calling VOP_RENAME(). ok? Index: nfs/nfs_serv.c === RCS file: /cvs/src/sys/nfs/nfs_serv.c,v retrieving revision 1.120 diff -u -p -r

Re: uvm_pagedequeue()

2022-05-12 Thread Martin Pieuchot
On 10/05/22(Tue) 20:23, Mark Kettenis wrote: > > Date: Tue, 10 May 2022 18:45:21 +0200 > > From: Martin Pieuchot > > > > On 05/05/22(Thu) 14:54, Martin Pieuchot wrote: > > > Diff below introduces a new wrapper to manipulate active/inactive page > > >

Re: uvm_pagedequeue()

2022-05-10 Thread Martin Pieuchot
On 05/05/22(Thu) 14:54, Martin Pieuchot wrote: > Diff below introduces a new wrapper to manipulate active/inactive page > queues. > > ok? Anyone? > Index: uvm/uvm_page.c > === > RCS file: /cvs/src/sys/uvm/uvm_

Re: uvm: Consider BUFPAGES_DEFICIT in swap_shortage

2022-05-09 Thread Martin Pieuchot
On 05/05/22(Thu) 10:56, Bob Beck wrote: > On Thu, May 05, 2022 at 10:16:23AM -0600, Bob Beck wrote: > > Ugh. You???re digging in the most perilous parts of the pile. > > > > I will go look with you??? sigh. (This is not yet an ok for that.) > > > > > On May

uvm: Consider BUFPAGES_DEFICIT in swap_shortage

2022-05-05 Thread Martin Pieuchot
When considering the amount of free pages in the page daemon a small amount is always kept for the buffer cache... except in one place. The diff below gets rid of this exception. This is important because uvmpd_scan() is called conditionally using the following check: if (uvmexp.free - BUFPA

uvm_pagedequeue()

2022-05-05 Thread Martin Pieuchot
Diff below introduces a new wrapper to manipulate active/inactive page queues. ok? Index: uvm/uvm_page.c === RCS file: /cvs/src/sys/uvm/uvm_page.c,v retrieving revision 1.165 diff -u -p -r1.165 uvm_page.c --- uvm/uvm_page.c 4 M

Merge swap-backed and object-backed inactive lists

2022-05-02 Thread Martin Pieuchot
Let's simplify the existing logic and use a single list for inactive pages. uvmpd_scan_inactive() already does a lot of check if it finds a page which is swap-backed. This will be improved in a next change. ok? Index: uvm/uvm.h ===

uvmpd_scan(): Recheck PG_BUSY after locking the page

2022-04-28 Thread Martin Pieuchot
rw_enter(9) can sleep. When the lock is finally acquired by the pagedaemon the previous check might no longer be true and the page could be busy. In this case we shouldn't touch it. Diff below recheck for PG_BUSY after acquiring the lock and also use a variable for the lock to reduce the differe

Call uvm_pageactivate() from uvm_pageunwire()

2022-04-26 Thread Martin Pieuchot
I'd like to use a proper interface to add/remove pages on the active/inactive queues. This will help for lock assertions and help improving the existing LRU limitations. Diff below makes uvm_pageunwire() call uvm_pageactivate() instead of inserting the page itself. ok? Index: uvm/uvm_page.c ===

Decrement uvmexp.swpgonly

2022-04-26 Thread Martin Pieuchot
Small diff to decrement the counter only if the I/O succeed. This prevent a false positive if a check is performed before an error is returned. ok? Index: uvm/uvm_swap.c === RCS file: /cvs/src/sys/uvm/uvm_swap.c,v retrieving revisio

Re: refcount btrace

2022-04-11 Thread Martin Pieuchot
On 08/04/22(Fri) 12:16, Alexander Bluhm wrote: > On Fri, Apr 08, 2022 at 02:39:34AM +, Visa Hankala wrote: > > On Thu, Apr 07, 2022 at 07:55:11PM +0200, Alexander Bluhm wrote: > > > On Wed, Mar 23, 2022 at 06:13:27PM +0100, Alexander Bluhm wrote: > > > > In my opinion tracepoints give insight a

Kill selrecord()

2022-04-11 Thread Martin Pieuchot
Now that poll(2) & select(2) use the kqueue backend under the hood we can start retiring the old machinery. The diff below does not touch driver definitions, however it : - kills selrecord() & doselwakeup() - make it obvious that `kern.nselcoll' is now always 0 - Change all poll/select hooks

Re: refcount btrace

2022-03-21 Thread Martin Pieuchot
On 20/03/22(Sun) 05:39, Visa Hankala wrote: > On Sat, Mar 19, 2022 at 12:10:11AM +0100, Alexander Bluhm wrote: > > On Thu, Mar 17, 2022 at 07:25:27AM +, Visa Hankala wrote: > > > On Thu, Mar 17, 2022 at 12:42:13AM +0100, Alexander Bluhm wrote: > > > > I would like to use btrace to debug refernc

Swap encrypt under memory pressure

2022-03-12 Thread Martin Pieuchot
Try to allocate the buffer before doing the encryption, if it fails we do not spend time doing the encryption. This reduce the pressure when swapping with low memory. ok? Index: uvm/uvm_swap.c === RCS file: /cvs/src/sys/uvm/uvm_swap

Re: more MAKEDEV cleanup

2022-02-10 Thread Martin Pieuchot
On 05/04/21(Mon) 09:25, Miod Vallat wrote: > The following diff attempts to clean up a few loose ends in the current > MAKEDEV files: > > - remove no-longer applicable device definitions (MSCP and SMD disks, > this kind of thing). > - makes sure all platforms use the same `ramdisk' target for >

Re: uvm_unmap_kill_entry(): unwire with map lock held

2022-02-04 Thread Martin Pieuchot
On 04/02/22(Fri) 03:39, Klemens Nanni wrote: > [...] > ... with the lock grabbed in uvm_map_teardown() that is, otherwise > the first call path can lock against itself (regress/misc/posixtestsuite > is a reproduce for this): > > vm_map_lock_read_ln+0x38 > uvm_fault_unwire+0x58 >

Re: uvm_unmap_kill_entry(): unwire with map lock held

2022-01-31 Thread Martin Pieuchot
On 31/01/22(Mon) 10:24, Klemens Nanni wrote: > Running with my uvm assert diff showed that uvm_fault_unwire_locked() > was called without any locks held. > > This happened when I rebooted my machine: > > uvm_fault_unwire_locked() > uvm_unmap_kill_entry_withlock() > uvm_unmap_kil

Re: unlock mmap(2) for anonymous mappings

2022-01-24 Thread Martin Pieuchot
On 24/01/22(Mon) 12:06, Klemens Nanni wrote: > On Sun, Jan 16, 2022 at 09:22:50AM -0300, Martin Pieuchot wrote: > > IMHO this approach of let's try if it works now and revert if it isn't > > doesn't help us make progress. I'd be more confident seeing diffs th

Re: DDBPROF: move option to amd64,i386 GENERIC

2022-01-18 Thread Martin Pieuchot
On 18/01/22(Tue) 04:38, Klemens Nanni wrote: > While intended for more architectures, DDBPROF is strictly amd64 and > i386 only, so the machine-independent sys/conf/GENERIC does not seem fit > (until all architectures are supported). This define should die. There's no need to polish this turd. S

Re: uvm_swap: introduce uvm_swap_data_lock

2022-01-16 Thread Martin Pieuchot
Nice! On 30/12/21(Thu) 23:38, Theo Buehler wrote: > The diff below does two things: it adds a uvm_swap_data_lock mutex and > trades it for the KERNEL_LOCK in uvm_swapisfull() and uvm_swap_markbad() Why is it enough? Which fields is the lock protecting in these function? Is it `uvmexp.swpages',

Re: unlock mmap(2) for anonymous mappings

2022-01-16 Thread Martin Pieuchot
On 14/01/22(Fri) 23:01, Mark Kettenis wrote: > > Date: Tue, 11 Jan 2022 23:13:20 + > > From: Klemens Nanni > > > > On Tue, Jan 11, 2022 at 09:54:44AM -0700, Theo de Raadt wrote: > > > > Now this is clearly a "slow" path. I don't think there is any reason > > > > not to put all the code in th

Re: patch: add a new ktrace facility for replacing some printf-debug

2022-01-07 Thread Martin Pieuchot
On 07/01/22(Fri) 10:54, Sebastien Marie wrote: > Hi, > > Debugging some code paths is complex: for example, unveil(2) code is > involved inside VFS, and using DEBUG_UNVEIL means that the kernel is > spamming printf() for all processes using unveil(2) (a lot of > processes) instead of just the inte

Re: gprof: Profiling a multi-threaded application

2021-12-10 Thread Martin Pieuchot
On 10/12/21(Fri) 09:56, Yuichiro NAITO wrote: > Any comments about this topic? I'm ok with this approach. I would appreciate if somebody else could take it over, I'm too busy with other stuff. > On 7/12/21 18:09, Yuichiro NAITO wrote: > > Hi, Martin > > > > n

Re: net write in pcb hash

2021-12-09 Thread Martin Pieuchot
On 08/12/21(Wed) 22:39, Alexander Bluhm wrote: > On Wed, Dec 08, 2021 at 03:28:34PM -0300, Martin Pieuchot wrote: > > On 04/12/21(Sat) 01:02, Alexander Bluhm wrote: > > > Hi, > > > > > > As I want a read-only net lock for forwarding, all paths should be > &

  1   2   3   4   5   6   7   8   9   10   >