Unkillable process in "vm map (user)"
I was experimenting with ports/devel/libmill (which is a library that
provides Go-styly functionality for C programs) and managed to create
an unkillable process by spawning 100 "goroutines" (think very
cheap "thread" or "coroutine") joined by "channels" (think message
passing pipes). (The program ran basically instantaneously with 1
or 10 "goroutines", and the Go version has no problems with 100
goroutines on a much smaller system).
According to SIGINFO, it's blocked on "vm map (user)" but I can't kill
it. Can anyone suggest a way to unwedge it?
This is on a system running FreeBSD/amd64 11.1-STABLE r324494.
server% procstat -kk 452
PIDTID COMMTDNAME KSTACK
452 102382 chain - mi_switch+0x17c
sleepq_switch+0x118 sleepq_wait+0x43 _sx_slock_hard+0x34e _sx_slock+0xd4
vm_map_lookup+0xbd vm_fault_hold+0x194b vm_fault+0x75 trap_pfault+0x107
trap+0x382 calltrap+0x8
server% ps -wal -p 452
UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND
204 452 53567 0 20 0 244064932 2180 vm map ( DL+ 13 0:10.31 ./chain
100
server% cat src/mill/chain.c
#include
#include
#include
coroutine void f(chan left, chan right) {
chs(left, int, 1 + chr(right, int));
}
int main(int argc, char **argv) {
int i, n = argv[1] ? atoi(argv[1]) : 1;
chan leftmost = chmake(int, 0);
chan left = NULL;
chan right = leftmost;
for (i = 0; i < n; i++) {
left = right;
right = chmake(int, 0);
go(f(left, right));
}
chs(right, int, 0);
i = chr(leftmost, int);
printf("result = %d\n", i);
return 0;
}
server%
--
Peter Jeremy
signature.asc
Description: PGP signature
Re: Unkillable process in "vm map (user)"
On Mon, Dec 11, 2017 at 07:09:31AM +1100, Peter Jeremy wrote: > I was experimenting with ports/devel/libmill (which is a library that > provides Go-styly functionality for C programs) and managed to create > an unkillable process by spawning 100 "goroutines" (think very > cheap "thread" or "coroutine") joined by "channels" (think message > passing pipes). (The program ran basically instantaneously with 1 > or 10 "goroutines", and the Go version has no problems with 100 > goroutines on a much smaller system). > > According to SIGINFO, it's blocked on "vm map (user)" but I can't kill > it. Can anyone suggest a way to unwedge it? > > This is on a system running FreeBSD/amd64 11.1-STABLE r324494. Ensure that you use at least r326188. > > server% procstat -kk 452 > PIDTID COMMTDNAME KSTACK > 452 102382 chain - mi_switch+0x17c > sleepq_switch+0x118 sleepq_wait+0x43 _sx_slock_hard+0x34e _sx_slock+0xd4 > vm_map_lookup+0xbd vm_fault_hold+0x194b vm_fault+0x75 trap_pfault+0x107 > trap+0x382 calltrap+0x8 There is another thread owning the map lock, and seeing what that thread does is the next step. Can you provide a binary to reproduce which does not depend on any library except the base libs ? ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Unkillable process in "vm map (user)"
On Sun, Dec 10, 2017 at 10:42:17PM +0200, Konstantin Belousov wrote: > On Mon, Dec 11, 2017 at 07:09:31AM +1100, Peter Jeremy wrote: > > I was experimenting with ports/devel/libmill (which is a library that > > provides Go-styly functionality for C programs) and managed to create > > an unkillable process by spawning 100 "goroutines" (think very > > cheap "thread" or "coroutine") joined by "channels" (think message > > passing pipes). (The program ran basically instantaneously with 1 > > or 10 "goroutines", and the Go version has no problems with 100 > > goroutines on a much smaller system). > > > > According to SIGINFO, it's blocked on "vm map (user)" but I can't kill > > it. Can anyone suggest a way to unwedge it? > > > > This is on a system running FreeBSD/amd64 11.1-STABLE r324494. > Ensure that you use at least r326188. > > > > > server% procstat -kk 452 > > PIDTID COMMTDNAME KSTACK > > 452 102382 chain - mi_switch+0x17c > > sleepq_switch+0x118 sleepq_wait+0x43 _sx_slock_hard+0x34e _sx_slock+0xd4 > > vm_map_lookup+0xbd vm_fault_hold+0x194b vm_fault+0x75 trap_pfault+0x107 > > trap+0x382 calltrap+0x8 > > There is another thread owning the map lock, and seeing what that thread > does is the next step. > > Can you provide a binary to reproduce which does not depend on any > library except the base libs ? Here's some more info, using the original scenario: https://people.freebsd.org/~pho/stress/log/kostik1070.txt - Peter ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Unkillable process in "vm map (user)"
On Fri, Dec 22, 2017 at 10:26:07AM +0100, Peter Holm wrote:
> Here's some more info, using the original scenario:
> https://people.freebsd.org/~pho/stress/log/kostik1070.txt
This is somewhat weird but also not too puzzling.
The vmdaemon (pid 41) is running, it tries to reduce the count of resident
pages in some pmap, most likely the one from the pid 20655. This process
seems to be huge: according to the v_stats, there is 15681264 inactive pages,
and the pagedaemon tries to obtain a vm object lock which is owned by
vmdaemon, resident count for that object is 15897170 (~64Gb).
So basically almost all memory belongs to the single object and vmdaemon
processing it. Since the object' queue is huge, the map and the object
locks are taken for long time, preventing other processes touching them
from making a progress.
Might be try this (it combines new changes with the OOM patch). I am not
sure that should_yield() in the vm_swapout_object_deactivate_pages() is
a good idea unconditionally, but it might be better than the current
situation.
diff --git a/sys/vm/vm_fault.c b/sys/vm/vm_fault.c
index ece496407c2..ce6208569c6 100644
--- a/sys/vm/vm_fault.c
+++ b/sys/vm/vm_fault.c
@@ -134,6 +134,16 @@ static void vm_fault_dontneed(const struct faultstate *fs,
vm_offset_t vaddr,
static void vm_fault_prefault(const struct faultstate *fs, vm_offset_t addra,
int backward, int forward);
+static int vm_pfault_oom_attempts = 3;
+SYSCTL_INT(_vm, OID_AUTO, pfault_oom_attempts, CTLFLAG_RWTUN,
+&vm_pfault_oom_attempts, 0,
+"");
+
+static int vm_pfault_oom_wait = 10;
+SYSCTL_INT(_vm, OID_AUTO, pfault_oom_wait, CTLFLAG_RWTUN,
+&vm_pfault_oom_wait, 0,
+"");
+
static inline void
release_page(struct faultstate *fs)
{
@@ -531,7 +541,7 @@ vm_fault_hold(vm_map_t map, vm_offset_t vaddr, vm_prot_t
fault_type,
vm_pindex_t retry_pindex;
vm_prot_t prot, retry_prot;
int ahead, alloc_req, behind, cluster_offset, error, era, faultcount;
- int locked, nera, result, rv;
+ int locked, nera, oom, result, rv;
u_char behavior;
boolean_t wired;/* Passed by reference. */
bool dead, hardfault, is_first_object_locked;
@@ -542,7 +552,9 @@ vm_fault_hold(vm_map_t map, vm_offset_t vaddr, vm_prot_t
fault_type,
nera = -1;
hardfault = false;
-RetryFault:;
+RetryFault:
+ oom = 0;
+RetryFault_oom:
/*
* Find the backing store object and offset into it to begin the
@@ -787,7 +799,17 @@ RetryFault:;
}
if (fs.m == NULL) {
unlock_and_deallocate(&fs);
- VM_WAITPFAULT;
+ if (vm_pfault_oom_attempts < 0 ||
+ oom < vm_pfault_oom_attempts) {
+ oom++;
+ vm_waitpfault(vm_pfault_oom_wait);
+ goto RetryFault_oom;
+ }
+ if (bootverbose)
+ printf(
+ "proc %d (%s) failed to alloc page on fault, starting OOM\n",
+ curproc->p_pid, curproc->p_comm);
+ vm_pageout_oom(VM_OOM_MEM_PF);
goto RetryFault;
}
}
diff --git a/sys/vm/vm_page.c b/sys/vm/vm_page.c
index 0397dfef457..7d403497550 100644
--- a/sys/vm/vm_page.c
+++ b/sys/vm/vm_page.c
@@ -2652,7 +2652,7 @@ vm_page_reclaim_contig(int req, u_long npages, vm_paddr_t
low, vm_paddr_t high,
* - Called in various places before memory allocations.
*/
static void
-_vm_wait(void)
+_vm_wait(int timo)
{
mtx_assert(&vm_page_queue_free_mtx, MA_OWNED);
@@ -2669,16 +2669,16 @@ _vm_wait(void)
}
vm_pages_needed = true;
msleep(&vm_cnt.v_free_count, &vm_page_queue_free_mtx, PDROP |
PVM,
- "vmwait", 0);
+ "vmwait", timo * hz);
}
}
void
-vm_wait(void)
+vm_wait(int timo)
{
mtx_lock(&vm_page_queue_free_mtx);
- _vm_wait();
+ _vm_wait(timo);
}
/*
@@ -2703,7 +2703,7 @@ vm_page_alloc_fail(vm_object_t object, int req)
if (req & (VM_ALLOC_WAITOK | VM_ALLOC_WAITFAIL)) {
if (object != NULL)
VM_OBJECT_WUNLOCK(object);
- _vm_wait();
+ _vm_wait(0);
if (object != NULL)
VM_OBJECT_WLOCK(object);
if (req & VM_ALLOC_WAITOK)
@@ -2724,7 +2724,7 @@ vm_page_alloc_fail(vm_object_t object, int req)
* this balance without careful testing first.
*/
void
-vm_waitpfault(void)
+vm_waitpfault(int timo)
{
mtx_lock(&vm_page_queue_free_mtx);
@@ -2734,7 +2734,7 @@ vm_waitpfault(void)
}
vm_pages_needed = tr
Re: Unkillable process in "vm map (user)"
On Fri, Dec 22, 2017 at 02:45:21PM +0200, Konstantin Belousov wrote: > On Fri, Dec 22, 2017 at 10:26:07AM +0100, Peter Holm wrote: > > Here's some more info, using the original scenario: > > https://people.freebsd.org/~pho/stress/log/kostik1070.txt > > This is somewhat weird but also not too puzzling. > > The vmdaemon (pid 41) is running, it tries to reduce the count of resident > pages in some pmap, most likely the one from the pid 20655. This process > seems to be huge: according to the v_stats, there is 15681264 inactive pages, > and the pagedaemon tries to obtain a vm object lock which is owned by > vmdaemon, resident count for that object is 15897170 (~64Gb). > > So basically almost all memory belongs to the single object and vmdaemon > processing it. Since the object' queue is huge, the map and the object > locks are taken for long time, preventing other processes touching them > from making a progress. > > Might be try this (it combines new changes with the OOM patch). I am not > sure that should_yield() in the vm_swapout_object_deactivate_pages() is > a good idea unconditionally, but it might be better than the current > situation. > > diff --git a/sys/vm/vm_fault.c b/sys/vm/vm_fault.c > index ece496407c2..ce6208569c6 100644 > --- a/sys/vm/vm_fault.c The patch fixes the problem I got with this scenario. - Peter ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
