Module Name: src
Committed By: martin
Date: Tue Feb 27 09:07:33 UTC 2018
Modified Files:
src/sys/arch/alpha/alpha [netbsd-8]: pmap.c
src/sys/arch/m68k/m68k [netbsd-8]: pmap_motorola.c
src/sys/arch/powerpc/oea [netbsd-8]: pmap.c
src/sys/arch/sparc64/sparc64 [netbsd-8]: pmap.c
src/sys/arch/x86/x86 [netbsd-8]: pmap.c
src/sys/dev/dtv [netbsd-8]: dtv_scatter.c
src/sys/dev/marvell [netbsd-8]: mvxpsec.c
src/sys/kern [netbsd-8]: subr_extent.c subr_pool.c uipc_mbuf.c
src/sys/opencrypto [netbsd-8]: crypto.c
src/sys/sys [netbsd-8]: mbuf.h pool.h
src/sys/ufs/chfs [netbsd-8]: chfs_malloc.c
src/sys/uvm [netbsd-8]: uvm_fault.c
Log Message:
Pull up following revision(s) (requested by mrg in ticket #593):
sys/dev/marvell/mvxpsec.c: revision 1.2
sys/arch/m68k/m68k/pmap_motorola.c: revision 1.70
sys/opencrypto/crypto.c: revision 1.102
sys/arch/sparc64/sparc64/pmap.c: revision 1.308
sys/ufs/chfs/chfs_malloc.c: revision 1.5
sys/arch/powerpc/oea/pmap.c: revision 1.95
sys/sys/pool.h: revision 1.80,1.82
sys/kern/subr_pool.c: revision 1.209-1.216,1.219-1.220
sys/arch/alpha/alpha/pmap.c: revision 1.262
sys/kern/uipc_mbuf.c: revision 1.173
sys/uvm/uvm_fault.c: revision 1.202
sys/sys/mbuf.h: revision 1.172
sys/kern/subr_extent.c: revision 1.86
sys/arch/x86/x86/pmap.c: revision 1.266 (via patch)
sys/dev/dtv/dtv_scatter.c: revision 1.4
Allow only one pending call to a pool's backing allocator at a time.
Candidate fix for problems with hanging after kva fragmentation related
to PR kern/45718.
Proposed on tech-kern:
https://mail-index.NetBSD.org/tech-kern/2017/10/23/msg022472.html
Tested by bouyer@ on i386.
This makes one small change to the semantics of pool_prime and
pool_setlowat: they may fail with EWOULDBLOCK instead of ENOMEM, if
there is a pending call to the backing allocator in another thread but
we are not actually out of memory. That is unlikely because nearly
always these are used during initialization, when the pool is not in
use.
Define the new flag too for previous commit.
pool_grow can now fail even when sleeping is ok. Catch this case in pool_get
and retry.
Assert that pool_get failure happens only with PR_NOWAIT.
This would have caught the mistake I made last week leading to null
pointer dereferences all over the place, a mistake which I evidently
poorly scheduled alongside maxv's change to the panic message on x86
for null pointer dereferences.
Since pr_lock is now used to wait for two things now (PR_GROWING and
PR_WANTED) we need to loop for the condition we wanted.
make the KASSERTMSG/panic strings consistent as '%s: [%s], __func__, wchan'
Handle the ERESTART case from pool_grow()
don't pass 0 to the pool flags
Guess pool_cache_get(pc, 0) means PR_WAITOK here.
Earlier on in the same context we use kmem_alloc(sz, KM_SLEEP).
use PR_WAITOK everywhere.
use PR_NOWAIT.
Don't use 0 for PR_NOWAIT
use PR_NOWAIT instead of 0
panic ex nihilo -- PR_NOWAITing for zerot
Add assertions that either PR_WAITOK or PR_NOWAIT are set.
- fix an assert; we can reach there if we are nowait or limitfail.
- when priming the pool and failing with ERESTART, don't decrement the number
of pages; this avoids the issue of returning an ERESTART when we get to 0,
and is more correct.
- simplify the pool_grow code, and don't wakeup things if we ENOMEM.
In pmap_enter_ma(), only try to allocate pves if we might need them,
and even if that fails, only fail the operation if we later discover
that we really do need them. This implements the requirement that
pmap_enter(PMAP_CANFAIL) must not fail when replacing an existing
mapping with the first mapping of a new page, which is an unintended
consequence of the changes from the rmind-uvmplock branch in 2011.
The problem arises when pmap_enter(PMAP_CANFAIL) is used to replace an existing
pmap mapping with a mapping of a different page (eg. to resolve a
copy-on-write).
If that fails and leaves the old pmap entry in place, then UVM won't hold
the right locks when it eventually retries. This entanglement of the UVM and
pmap locking was done in rmind-uvmplock in order to improve performance,
but it also means that the UVM state and pmap state need to be kept in sync
more than they did before. It would be possible to handle this in the UVM code
instead of in the pmap code, but these pmap changes improve the handling of
low memory situations in general, and handling this in UVM would be clunky,
so this seemed like the better way to go.
This somewhat indirectly fixes PR 52706, as well as the failing assertion
about "uvm_page_locked_p(old_pg)". (but only on x86, various other platforms
will need their own changes to handle this issue.)
In uvm_fault_upper_enter(), if pmap_enter(PMAP_CANFAIL) fails, assert that
the pmap did not leave around a now-stale pmap mapping for an old page.
If such a pmap mapping still existed after we unlocked the vm_map,
the UVM code would not know later that it would need to lock the
lower layer object while calling the pmap to remove or replace that
stale pmap mapping. See PR 52706 for further details.
hopefully workaround the irregularly "fork fails in init" problem.
if a pool is growing, and the grower is PR_NOWAIT, mark this.
if another caller wants to grow the pool and is also PR_NOWAIT,
busy-wait for the original caller, which should either succeed
or hard-fail fairly quickly.
implement the busy-wait by unlocking and relocking this pools
mutex and returning ERESTART. other methods (such as having
the caller do this) were significantly more code and this hack
is fairly localised.
ok chs@ riastradh@
Don't release the lock in the PR_NOWAIT allocation. Move flags setting
after the acquiring the mutex. (from Tobias Nygren)
apply the change from arch/x86/x86/pmap.c rev. 1.266 commitid vZRjvmxG7YTHLOfA:
In pmap_enter_ma(), only try to allocate pves if we might need them,
and even if that fails, only fail the operation if we later discover
that we really do need them. If we are replacing an existing mapping,
reuse the pv structure where possible.
This implements the requirement that pmap_enter(PMAP_CANFAIL) must not fail
when replacing an existing mapping with the first mapping of a new page,
which is an unintended consequence of the changes from the rmind-uvmplock
branch in 2011.
The problem arises when pmap_enter(PMAP_CANFAIL) is used to replace an existing
pmap mapping with a mapping of a different page (eg. to resolve a
copy-on-write).
If that fails and leaves the old pmap entry in place, then UVM won't hold
the right locks when it eventually retries. This entanglement of the UVM and
pmap locking was done in rmind-uvmplock in order to improve performance,
but it also means that the UVM state and pmap state need to be kept in sync
more than they did before. It would be possible to handle this in the UVM code
instead of in the pmap code, but these pmap changes improve the handling of
low memory situations in general, and handling this in UVM would be clunky,
so this seemed like the better way to go.
This somewhat indirectly fixes PR 52706 on the remaining platforms where
this problem existed.
To generate a diff of this commit:
cvs rdiff -u -r1.261 -r1.261.8.1 src/sys/arch/alpha/alpha/pmap.c
cvs rdiff -u -r1.69 -r1.69.8.1 src/sys/arch/m68k/m68k/pmap_motorola.c
cvs rdiff -u -r1.94 -r1.94.8.1 src/sys/arch/powerpc/oea/pmap.c
cvs rdiff -u -r1.307 -r1.307.6.1 src/sys/arch/sparc64/sparc64/pmap.c
cvs rdiff -u -r1.245.6.1 -r1.245.6.2 src/sys/arch/x86/x86/pmap.c
cvs rdiff -u -r1.3 -r1.3.2.1 src/sys/dev/dtv/dtv_scatter.c
cvs rdiff -u -r1.1 -r1.1.12.1 src/sys/dev/marvell/mvxpsec.c
cvs rdiff -u -r1.80 -r1.80.8.1 src/sys/kern/subr_extent.c
cvs rdiff -u -r1.207 -r1.207.6.1 src/sys/kern/subr_pool.c
cvs rdiff -u -r1.172 -r1.172.6.1 src/sys/kern/uipc_mbuf.c
cvs rdiff -u -r1.78.2.4 -r1.78.2.5 src/sys/opencrypto/crypto.c
cvs rdiff -u -r1.170.2.1 -r1.170.2.2 src/sys/sys/mbuf.h
cvs rdiff -u -r1.79 -r1.79.10.1 src/sys/sys/pool.h
cvs rdiff -u -r1.4 -r1.4.30.1 src/sys/ufs/chfs/chfs_malloc.c
cvs rdiff -u -r1.199.6.2 -r1.199.6.3 src/sys/uvm/uvm_fault.c
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
Modified files:
Index: src/sys/arch/alpha/alpha/pmap.c
diff -u src/sys/arch/alpha/alpha/pmap.c:1.261 src/sys/arch/alpha/alpha/pmap.c:1.261.8.1
--- src/sys/arch/alpha/alpha/pmap.c:1.261 Fri Dec 23 07:15:27 2016
+++ src/sys/arch/alpha/alpha/pmap.c Tue Feb 27 09:07:33 2018
@@ -1,4 +1,4 @@
-/* $NetBSD: pmap.c,v 1.261 2016/12/23 07:15:27 cherry Exp $ */
+/* $NetBSD: pmap.c,v 1.261.8.1 2018/02/27 09:07:33 martin Exp $ */
/*-
* Copyright (c) 1998, 1999, 2000, 2001, 2007, 2008 The NetBSD Foundation, Inc.
@@ -140,7 +140,7 @@
#include <sys/cdefs.h> /* RCS ID & Copyright macro defns */
-__KERNEL_RCSID(0, "$NetBSD: pmap.c,v 1.261 2016/12/23 07:15:27 cherry Exp $");
+__KERNEL_RCSID(0, "$NetBSD: pmap.c,v 1.261.8.1 2018/02/27 09:07:33 martin Exp $");
#include <sys/param.h>
#include <sys/systm.h>
@@ -439,7 +439,8 @@ static struct pool_cache pmap_tlb_shootd
* Internal routines
*/
static void alpha_protection_init(void);
-static bool pmap_remove_mapping(pmap_t, vaddr_t, pt_entry_t *, bool, long);
+static bool pmap_remove_mapping(pmap_t, vaddr_t, pt_entry_t *, bool, long,
+ pv_entry_t *);
static void pmap_changebit(struct vm_page *, pt_entry_t, pt_entry_t, long);
/*
@@ -466,8 +467,9 @@ static int pmap_l1pt_ctor(void *, void *
* PV table management functions.
*/
static int pmap_pv_enter(pmap_t, struct vm_page *, vaddr_t, pt_entry_t *,
- bool);
-static void pmap_pv_remove(pmap_t, struct vm_page *, vaddr_t, bool);
+ bool, pv_entry_t);
+static void pmap_pv_remove(pmap_t, struct vm_page *, vaddr_t, bool,
+ pv_entry_t *);
static void *pmap_pv_page_alloc(struct pool *, int);
static void pmap_pv_page_free(struct pool *, void *);
@@ -1266,7 +1268,7 @@ pmap_remove(pmap_t pmap, vaddr_t sva, va
sva);
#endif
needisync |= pmap_remove_mapping(pmap, sva,
- l3pte, true, cpu_id);
+ l3pte, true, cpu_id, NULL);
}
sva += PAGE_SIZE;
}
@@ -1343,7 +1345,7 @@ pmap_remove(pmap_t pmap, vaddr_t sva, va
pmap_remove_mapping(
pmap, sva,
l3pte, true,
- cpu_id);
+ cpu_id, NULL);
}
/*
@@ -1450,7 +1452,7 @@ pmap_page_protect(struct vm_page *pg, vm
panic("pmap_page_protect: bad mapping");
#endif
if (pmap_remove_mapping(pmap, pv->pv_va, pv->pv_pte,
- false, cpu_id) == true) {
+ false, cpu_id, NULL)) {
if (pmap == pmap_kernel())
needkisync |= true;
else
@@ -1558,6 +1560,7 @@ pmap_enter(pmap_t pmap, vaddr_t va, padd
{
struct vm_page *pg; /* if != NULL, managed page */
pt_entry_t *pte, npte, opte;
+ pv_entry_t opv = NULL;
paddr_t opa;
bool tflush = true;
bool hadasm = false; /* XXX gcc -Wuninitialized */
@@ -1750,14 +1753,15 @@ pmap_enter(pmap_t pmap, vaddr_t va, padd
*/
pmap_physpage_addref(pte);
}
- needisync |= pmap_remove_mapping(pmap, va, pte, true, cpu_id);
+ needisync |= pmap_remove_mapping(pmap, va, pte, true, cpu_id, &opv);
validate_enterpv:
/*
* Enter the mapping into the pv_table if appropriate.
*/
if (pg != NULL) {
- error = pmap_pv_enter(pmap, pg, va, pte, true);
+ error = pmap_pv_enter(pmap, pg, va, pte, true, opv);
+ opv = NULL;
if (error) {
pmap_l3pt_delref(pmap, va, pte, cpu_id);
if (flags & PMAP_CANFAIL)
@@ -1845,6 +1849,8 @@ pmap_enter(pmap_t pmap, vaddr_t va, padd
out:
PMAP_UNLOCK(pmap);
PMAP_MAP_TO_HEAD_UNLOCK();
+ if (opv)
+ pmap_pv_free(opv);
return error;
}
@@ -2422,7 +2428,7 @@ alpha_protection_init(void)
*/
static bool
pmap_remove_mapping(pmap_t pmap, vaddr_t va, pt_entry_t *pte,
- bool dolock, long cpu_id)
+ bool dolock, long cpu_id, pv_entry_t *opvp)
{
paddr_t pa;
struct vm_page *pg; /* if != NULL, page is managed */
@@ -2434,8 +2440,8 @@ pmap_remove_mapping(pmap_t pmap, vaddr_t
#ifdef DEBUG
if (pmapdebug & (PDB_FOLLOW|PDB_REMOVE|PDB_PROTECT))
- printf("pmap_remove_mapping(%p, %lx, %p, %d, %ld)\n",
- pmap, va, pte, dolock, cpu_id);
+ printf("pmap_remove_mapping(%p, %lx, %p, %d, %ld, %p)\n",
+ pmap, va, pte, dolock, cpu_id, opvp);
#endif
/*
@@ -2511,7 +2517,8 @@ pmap_remove_mapping(pmap_t pmap, vaddr_t
*/
pg = PHYS_TO_VM_PAGE(pa);
KASSERT(pg != NULL);
- pmap_pv_remove(pmap, pg, va, dolock);
+ pmap_pv_remove(pmap, pg, va, dolock, opvp);
+ KASSERT(opvp == NULL || *opvp != NULL);
return (needisync);
}
@@ -2765,18 +2772,19 @@ vtophys(vaddr_t vaddr)
*/
static int
pmap_pv_enter(pmap_t pmap, struct vm_page *pg, vaddr_t va, pt_entry_t *pte,
- bool dolock)
+ bool dolock, pv_entry_t newpv)
{
struct vm_page_md * const md = VM_PAGE_TO_MD(pg);
- pv_entry_t newpv;
kmutex_t *lock;
/*
* Allocate and fill in the new pv_entry.
*/
- newpv = pmap_pv_alloc();
- if (newpv == NULL)
- return ENOMEM;
+ if (newpv == NULL) {
+ newpv = pmap_pv_alloc();
+ if (newpv == NULL)
+ return ENOMEM;
+ }
newpv->pv_va = va;
newpv->pv_pmap = pmap;
newpv->pv_pte = pte;
@@ -2820,7 +2828,8 @@ pmap_pv_enter(pmap_t pmap, struct vm_pag
* Remove a physical->virtual entry from the pv_table.
*/
static void
-pmap_pv_remove(pmap_t pmap, struct vm_page *pg, vaddr_t va, bool dolock)
+pmap_pv_remove(pmap_t pmap, struct vm_page *pg, vaddr_t va, bool dolock,
+ pv_entry_t *opvp)
{
struct vm_page_md * const md = VM_PAGE_TO_MD(pg);
pv_entry_t pv, *pvp;
@@ -2852,7 +2861,10 @@ pmap_pv_remove(pmap_t pmap, struct vm_pa
mutex_exit(lock);
}
- pmap_pv_free(pv);
+ if (opvp != NULL)
+ *opvp = pv;
+ else
+ pmap_pv_free(pv);
}
/*
Index: src/sys/arch/m68k/m68k/pmap_motorola.c
diff -u src/sys/arch/m68k/m68k/pmap_motorola.c:1.69 src/sys/arch/m68k/m68k/pmap_motorola.c:1.69.8.1
--- src/sys/arch/m68k/m68k/pmap_motorola.c:1.69 Fri Dec 23 07:15:27 2016
+++ src/sys/arch/m68k/m68k/pmap_motorola.c Tue Feb 27 09:07:32 2018
@@ -1,4 +1,4 @@
-/* $NetBSD: pmap_motorola.c,v 1.69 2016/12/23 07:15:27 cherry Exp $ */
+/* $NetBSD: pmap_motorola.c,v 1.69.8.1 2018/02/27 09:07:32 martin Exp $ */
/*-
* Copyright (c) 1999 The NetBSD Foundation, Inc.
@@ -119,7 +119,7 @@
#include "opt_m68k_arch.h"
#include <sys/cdefs.h>
-__KERNEL_RCSID(0, "$NetBSD: pmap_motorola.c,v 1.69 2016/12/23 07:15:27 cherry Exp $");
+__KERNEL_RCSID(0, "$NetBSD: pmap_motorola.c,v 1.69.8.1 2018/02/27 09:07:32 martin Exp $");
#include <sys/param.h>
#include <sys/systm.h>
@@ -306,7 +306,8 @@ pa_to_pvh(paddr_t pa)
/*
* Internal routines
*/
-void pmap_remove_mapping(pmap_t, vaddr_t, pt_entry_t *, int);
+void pmap_remove_mapping(pmap_t, vaddr_t, pt_entry_t *, int,
+ struct pv_entry **);
bool pmap_testbit(paddr_t, int);
bool pmap_changebit(paddr_t, int, int);
int pmap_enter_ptpage(pmap_t, vaddr_t, bool);
@@ -843,7 +844,7 @@ pmap_remove(pmap_t pmap, vaddr_t sva, va
}
firstpage = false;
#endif
- pmap_remove_mapping(pmap, sva, pte, flags);
+ pmap_remove_mapping(pmap, sva, pte, flags, NULL);
}
pte++;
sva += PAGE_SIZE;
@@ -929,7 +930,7 @@ pmap_page_protect(struct vm_page *pg, vm
panic("pmap_page_protect: bad mapping");
#endif
pmap_remove_mapping(pv->pv_pmap, pv->pv_va,
- pte, PRM_TFLUSH|PRM_CFLUSH);
+ pte, PRM_TFLUSH|PRM_CFLUSH, NULL);
}
splx(s);
}
@@ -1048,6 +1049,7 @@ int
pmap_enter(pmap_t pmap, vaddr_t va, paddr_t pa, vm_prot_t prot, u_int flags)
{
pt_entry_t *pte;
+ struct pv_entry *opv = NULL;
int npte;
paddr_t opa;
bool cacheable = true;
@@ -1130,7 +1132,7 @@ pmap_enter(pmap_t pmap, vaddr_t va, padd
PMAP_DPRINTF(PDB_ENTER,
("enter: removing old mapping %lx\n", va));
pmap_remove_mapping(pmap, va, pte,
- PRM_TFLUSH|PRM_CFLUSH|PRM_KEEPPTPAGE);
+ PRM_TFLUSH|PRM_CFLUSH|PRM_KEEPPTPAGE, &opv);
}
/*
@@ -1179,7 +1181,12 @@ pmap_enter(pmap_t pmap, vaddr_t va, padd
if (pmap == npv->pv_pmap && va == npv->pv_va)
panic("pmap_enter: already in pv_tab");
#endif
- npv = pmap_alloc_pv();
+ if (opv != NULL) {
+ npv = opv;
+ opv = NULL;
+ } else {
+ npv = pmap_alloc_pv();
+ }
KASSERT(npv != NULL);
npv->pv_va = va;
npv->pv_pmap = pmap;
@@ -1346,6 +1353,9 @@ validate:
pmap_check_wiring("enter", trunc_page((vaddr_t)pte));
#endif
+ if (opv != NULL)
+ pmap_free_pv(opv);
+
return 0;
}
@@ -1659,7 +1669,7 @@ pmap_collect1(pmap_t pmap, paddr_t start
(void) pmap_extract(pmap, pv->pv_va, &kpa);
pmap_remove_mapping(pmap, pv->pv_va, NULL,
- PRM_TFLUSH|PRM_CFLUSH);
+ PRM_TFLUSH|PRM_CFLUSH, NULL);
/*
* Use the physical address to locate the original
@@ -1970,11 +1980,12 @@ pmap_prefer(vaddr_t foff, vaddr_t *vap)
*/
/* static */
void
-pmap_remove_mapping(pmap_t pmap, vaddr_t va, pt_entry_t *pte, int flags)
+pmap_remove_mapping(pmap_t pmap, vaddr_t va, pt_entry_t *pte, int flags,
+ struct pv_entry **opvp)
{
paddr_t pa;
struct pv_header *pvh;
- struct pv_entry *pv, *npv;
+ struct pv_entry *pv, *npv, *opv = NULL;
struct pmap *ptpmap;
st_entry_t *ste;
int s, bits;
@@ -1983,8 +1994,8 @@ pmap_remove_mapping(pmap_t pmap, vaddr_t
#endif
PMAP_DPRINTF(PDB_FOLLOW|PDB_REMOVE|PDB_PROTECT,
- ("pmap_remove_mapping(%p, %lx, %p, %x)\n",
- pmap, va, pte, flags));
+ ("pmap_remove_mapping(%p, %lx, %p, %x, %p)\n",
+ pmap, va, pte, flags, opvp));
/*
* PTE not provided, compute it from pmap and va.
@@ -2093,7 +2104,7 @@ pmap_remove_mapping(pmap_t pmap, vaddr_t
ptppv->pv_next);
#endif
pmap_remove_mapping(pmap_kernel(), ptpva,
- NULL, PRM_TFLUSH|PRM_CFLUSH);
+ NULL, PRM_TFLUSH|PRM_CFLUSH, NULL);
mutex_enter(uvm_kernel_object->vmobjlock);
uvm_pagefree(PHYS_TO_VM_PAGE(ptppa));
mutex_exit(uvm_kernel_object->vmobjlock);
@@ -2133,7 +2144,7 @@ pmap_remove_mapping(pmap_t pmap, vaddr_t
npv = pv->pv_next;
if (npv) {
*pv = *npv;
- pmap_free_pv(npv);
+ opv = npv;
} else
pv->pv_pmap = NULL;
} else {
@@ -2149,7 +2160,7 @@ pmap_remove_mapping(pmap_t pmap, vaddr_t
ste = npv->pv_ptste;
ptpmap = npv->pv_ptpmap;
pv->pv_next = npv->pv_next;
- pmap_free_pv(npv);
+ opv = npv;
pvh = pa_to_pvh(pa);
pv = &pvh->pvh_first;
}
@@ -2255,6 +2266,11 @@ pmap_remove_mapping(pmap_t pmap, vaddr_t
pvh->pvh_attrs |= bits;
splx(s);
+
+ if (opvp != NULL)
+ *opvp = opv;
+ else if (opv != NULL)
+ pmap_free_pv(opv);
}
/*
Index: src/sys/arch/powerpc/oea/pmap.c
diff -u src/sys/arch/powerpc/oea/pmap.c:1.94 src/sys/arch/powerpc/oea/pmap.c:1.94.8.1
--- src/sys/arch/powerpc/oea/pmap.c:1.94 Fri Dec 23 07:15:28 2016
+++ src/sys/arch/powerpc/oea/pmap.c Tue Feb 27 09:07:33 2018
@@ -1,4 +1,4 @@
-/* $NetBSD: pmap.c,v 1.94 2016/12/23 07:15:28 cherry Exp $ */
+/* $NetBSD: pmap.c,v 1.94.8.1 2018/02/27 09:07:33 martin Exp $ */
/*-
* Copyright (c) 2001 The NetBSD Foundation, Inc.
* All rights reserved.
@@ -63,7 +63,7 @@
*/
#include <sys/cdefs.h>
-__KERNEL_RCSID(0, "$NetBSD: pmap.c,v 1.94 2016/12/23 07:15:28 cherry Exp $");
+__KERNEL_RCSID(0, "$NetBSD: pmap.c,v 1.94.8.1 2018/02/27 09:07:33 martin Exp $");
#define PMAP_NOOPNAMES
@@ -1551,6 +1551,13 @@ pmap_pvo_reclaim(struct pmap *pm)
return NULL;
}
+static struct pool *
+pmap_pvo_pl(struct pvo_entry *pvo)
+{
+
+ return PVO_MANAGED_P(pvo) ? &pmap_mpvo_pool : &pmap_upvo_pool;
+}
+
/*
* This returns whether this is the first mapping of a page.
*/
@@ -1616,9 +1623,10 @@ pmap_pvo_enter(pmap_t pm, struct pool *p
#endif
pmap_interrupts_restore(msr);
if (pvo) {
- pmap_pvo_free(pvo);
+ KASSERT(pmap_pvo_pl(pvo) == pl);
+ } else {
+ pvo = pool_get(pl, poolflags);
}
- pvo = pool_get(pl, poolflags);
KASSERT((vaddr_t)pvo < VM_MIN_KERNEL_ADDRESS);
#ifdef DEBUG
@@ -1822,7 +1830,7 @@ void
pmap_pvo_free(struct pvo_entry *pvo)
{
- pool_put(PVO_MANAGED_P(pvo) ? &pmap_mpvo_pool : &pmap_upvo_pool, pvo);
+ pool_put(pmap_pvo_pl(pvo), pvo);
}
void
Index: src/sys/arch/sparc64/sparc64/pmap.c
diff -u src/sys/arch/sparc64/sparc64/pmap.c:1.307 src/sys/arch/sparc64/sparc64/pmap.c:1.307.6.1
--- src/sys/arch/sparc64/sparc64/pmap.c:1.307 Fri Feb 10 23:26:23 2017
+++ src/sys/arch/sparc64/sparc64/pmap.c Tue Feb 27 09:07:32 2018
@@ -1,4 +1,4 @@
-/* $NetBSD: pmap.c,v 1.307 2017/02/10 23:26:23 palle Exp $ */
+/* $NetBSD: pmap.c,v 1.307.6.1 2018/02/27 09:07:32 martin Exp $ */
/*
*
* Copyright (C) 1996-1999 Eduardo Horvath.
@@ -26,7 +26,7 @@
*/
#include <sys/cdefs.h>
-__KERNEL_RCSID(0, "$NetBSD: pmap.c,v 1.307 2017/02/10 23:26:23 palle Exp $");
+__KERNEL_RCSID(0, "$NetBSD: pmap.c,v 1.307.6.1 2018/02/27 09:07:32 martin Exp $");
#undef NO_VCACHE /* Don't forget the locked TLB in dostart */
#define HWREF
@@ -135,7 +135,7 @@ struct pool_cache pmap_pv_cache;
pv_entry_t pmap_remove_pv(struct pmap *, vaddr_t, struct vm_page *);
void pmap_enter_pv(struct pmap *, vaddr_t, paddr_t, struct vm_page *,
- pv_entry_t);
+ pv_entry_t *);
void pmap_page_cache(struct pmap *, paddr_t, int);
/*
@@ -1783,13 +1783,13 @@ pmap_enter(struct pmap *pm, vaddr_t va,
pte_t tte;
int64_t data;
paddr_t opa = 0, ptp; /* XXX: gcc */
- pv_entry_t pvh, npv = NULL, freepv;
+ pv_entry_t pvh, opv = NULL, npv;
struct vm_page *pg, *opg, *ptpg;
int s, i, uncached = 0, error = 0;
int size = PGSZ_8K; /* PMAP_SZ_TO_TTE(pa); */
bool wired = (flags & PMAP_WIRED) != 0;
- bool wasmapped = FALSE;
- bool dopv = TRUE;
+ bool wasmapped = false;
+ bool dopv = true;
/*
* Is this part of the permanent mappings?
@@ -1797,14 +1797,12 @@ pmap_enter(struct pmap *pm, vaddr_t va,
KASSERT(pm != pmap_kernel() || va < INTSTACK || va > EINTSTACK);
KASSERT(pm != pmap_kernel() || va < kdata || va > ekdata);
- /* Grab a spare PV. */
- freepv = pool_cache_get(&pmap_pv_cache, PR_NOWAIT);
- if (__predict_false(freepv == NULL)) {
- if (flags & PMAP_CANFAIL)
- return (ENOMEM);
- panic("pmap_enter: no pv entries available");
- }
- freepv->pv_next = NULL;
+ /*
+ * Grab a spare PV. Keep going even if this fails since we don't
+ * yet know if we will need it.
+ */
+
+ npv = pool_cache_get(&pmap_pv_cache, PR_NOWAIT);
/*
* If a mapping at this address already exists, check if we're
@@ -1819,7 +1817,7 @@ pmap_enter(struct pmap *pm, vaddr_t va,
if (opa != pa) {
opg = PHYS_TO_VM_PAGE(opa);
if (opg != NULL) {
- npv = pmap_remove_pv(pm, va, opg);
+ opv = pmap_remove_pv(pm, va, opg);
}
}
}
@@ -1849,31 +1847,21 @@ pmap_enter(struct pmap *pm, vaddr_t va,
/*
* make sure we have a pv entry ready if we need one.
*/
- if (pvh->pv_pmap == NULL || (wasmapped && opa == pa)) {
- if (npv != NULL) {
- /* free it */
- npv->pv_next = freepv;
- freepv = npv;
- npv = NULL;
- }
- if (wasmapped && opa == pa) {
- dopv = FALSE;
- }
+ if (wasmapped && opa == pa) {
+ dopv = false;
} else if (npv == NULL) {
- /* use the pre-allocated pv */
- npv = freepv;
- freepv = freepv->pv_next;
+ npv = opv;
+ opv = NULL;
+ if (npv == NULL) {
+ mutex_exit(&pmap_lock);
+ error = ENOMEM;
+ goto out;
+ }
}
ENTER_STAT(managed);
} else {
ENTER_STAT(unmanaged);
- dopv = FALSE;
- if (npv != NULL) {
- /* free it */
- npv->pv_next = freepv;
- freepv = npv;
- npv = NULL;
- }
+ dopv = false;
}
#ifndef NO_VCACHE
@@ -1945,11 +1933,6 @@ pmap_enter(struct pmap *pm, vaddr_t va,
if (!pmap_get_page(&ptp)) {
mutex_exit(&pmap_lock);
if (flags & PMAP_CANFAIL) {
- if (npv != NULL) {
- /* free it */
- npv->pv_next = freepv;
- freepv = npv;
- }
error = ENOMEM;
goto out;
} else {
@@ -1966,7 +1949,7 @@ pmap_enter(struct pmap *pm, vaddr_t va,
pmap_free_page_noflush(ptp);
}
if (dopv) {
- pmap_enter_pv(pm, va, pa, pg, npv);
+ pmap_enter_pv(pm, va, pa, pg, &npv);
}
mutex_exit(&pmap_lock);
@@ -2039,11 +2022,11 @@ pmap_enter(struct pmap *pm, vaddr_t va,
/* We will let the fast mmu miss interrupt load the new translation */
pv_check();
out:
- /* Catch up on deferred frees. */
- for (; freepv != NULL; freepv = npv) {
- npv = freepv->pv_next;
- pool_cache_put(&pmap_pv_cache, freepv);
- }
+ if (opv)
+ pool_cache_put(&pmap_pv_cache, opv);
+ if (npv)
+ pool_cache_put(&pmap_pv_cache, npv);
+
return error;
}
@@ -3302,14 +3285,16 @@ ctx_free(struct pmap *pm, struct cpu_inf
* physical to virtual map table.
*
* We enter here with the pmap locked.
+ * The pv_entry_t in *npvp is replaced with NULL if this function
+ * uses it, otherwise the caller needs to free it.
*/
void
pmap_enter_pv(struct pmap *pmap, vaddr_t va, paddr_t pa, struct vm_page *pg,
- pv_entry_t npv)
+ pv_entry_t *npvp)
{
struct vm_page_md * const md = VM_PAGE_TO_MD(pg);
- pv_entry_t pvh;
+ pv_entry_t pvh, npv;
KASSERT(mutex_owned(&pmap_lock));
@@ -3327,7 +3312,6 @@ pmap_enter_pv(struct pmap *pmap, vaddr_t
PV_SETVA(pvh, va);
pvh->pv_pmap = pmap;
pvh->pv_next = NULL;
- KASSERT(npv == NULL);
} else {
if (pg->loan_count == 0 && !(pvh->pv_va & PV_ALIAS)) {
@@ -3352,6 +3336,8 @@ pmap_enter_pv(struct pmap *pmap, vaddr_t
DPRINTF(PDB_ENTER, ("pmap_enter: new pv: pmap %p va %lx\n",
pmap, va));
+ npv = *npvp;
+ *npvp = NULL;
npv->pv_pmap = pmap;
npv->pv_va = va & PV_VAMASK;
npv->pv_next = pvh->pv_next;
Index: src/sys/arch/x86/x86/pmap.c
diff -u src/sys/arch/x86/x86/pmap.c:1.245.6.1 src/sys/arch/x86/x86/pmap.c:1.245.6.2
--- src/sys/arch/x86/x86/pmap.c:1.245.6.1 Wed Jul 5 20:23:08 2017
+++ src/sys/arch/x86/x86/pmap.c Tue Feb 27 09:07:33 2018
@@ -1,4 +1,4 @@
-/* $NetBSD: pmap.c,v 1.245.6.1 2017/07/05 20:23:08 snj Exp $ */
+/* $NetBSD: pmap.c,v 1.245.6.2 2018/02/27 09:07:33 martin Exp $ */
/*-
* Copyright (c) 2008, 2010, 2016, 2017 The NetBSD Foundation, Inc.
@@ -171,7 +171,7 @@
*/
#include <sys/cdefs.h>
-__KERNEL_RCSID(0, "$NetBSD: pmap.c,v 1.245.6.1 2017/07/05 20:23:08 snj Exp $");
+__KERNEL_RCSID(0, "$NetBSD: pmap.c,v 1.245.6.2 2018/02/27 09:07:33 martin Exp $");
#include "opt_user_ldt.h"
#include "opt_lockdebug.h"
@@ -1727,6 +1727,20 @@ pmap_vpage_cpu_init(struct cpu_info *ci)
* p v _ e n t r y f u n c t i o n s
*/
+static bool
+pmap_pp_needs_pve(struct pmap_page *pp)
+{
+
+ /*
+ * Adding a pv entry for this page only needs to allocate a pv_entry
+ * structure if the page already has at least one pv entry,
+ * since the first pv entry is stored in the pmap_page.
+ */
+
+ return (pp->pp_flags & PP_EMBEDDED) != 0 ||
+ !LIST_EMPTY(&pp->pp_head.pvh_list);
+}
+
/*
* pmap_free_pvs: free a list of pv_entrys
*/
@@ -4101,15 +4115,20 @@ pmap_enter_ma(struct pmap *pmap, vaddr_t
new_pp = NULL;
}
- /* get pves. */
- new_pve = pool_cache_get(&pmap_pv_cache, PR_NOWAIT);
- new_sparepve = pool_cache_get(&pmap_pv_cache, PR_NOWAIT);
- if (new_pve == NULL || new_sparepve == NULL) {
- if (flags & PMAP_CANFAIL) {
- error = ENOMEM;
- goto out2;
- }
- panic("pmap_enter: pve allocation failed");
+ /*
+ * Try to get pves now if we might need them.
+ * Keep going even if we fail, since we will not actually need them
+ * if we are just changing the permissions on an existing mapping,
+ * but we won't know if that's the case until later.
+ */
+
+ bool needpves = pmap_pp_needs_pve(new_pp);
+ if (new_pp && needpves) {
+ new_pve = pool_cache_get(&pmap_pv_cache, PR_NOWAIT);
+ new_sparepve = pool_cache_get(&pmap_pv_cache, PR_NOWAIT);
+ } else {
+ new_pve = NULL;
+ new_sparepve = NULL;
}
kpreempt_disable();
@@ -4129,10 +4148,31 @@ pmap_enter_ma(struct pmap *pmap, vaddr_t
}
/*
- * update the pte.
+ * Check if there is an existing mapping. If we are now sure that
+ * we need pves and we failed to allocate them earlier, handle that.
+ * Caching the value of oldpa here is safe because only the mod/ref bits
+ * can change while the pmap is locked.
*/
ptep = &ptes[pl1_i(va)];
+ opte = *ptep;
+ bool have_oldpa = pmap_valid_entry(opte);
+ paddr_t oldpa = pmap_pte2pa(opte);
+
+ if (needpves && (!have_oldpa || oldpa != pa) &&
+ (new_pve == NULL || new_sparepve == NULL)) {
+ pmap_unmap_ptes(pmap, pmap2);
+ if (flags & PMAP_CANFAIL) {
+ error = ENOMEM;
+ goto out;
+ }
+ panic("%s: pve allocation failed", __func__);
+ }
+
+ /*
+ * update the pte.
+ */
+
do {
opte = *ptep;
@@ -4170,7 +4210,7 @@ pmap_enter_ma(struct pmap *pmap, vaddr_t
*/
pmap_stats_update_bypte(pmap, npte, opte);
- if (ptp != NULL && !pmap_valid_entry(opte)) {
+ if (ptp != NULL && !have_oldpa) {
ptp->wire_count++;
}
KASSERT(ptp == NULL || ptp->wire_count > 1);
@@ -4189,16 +4229,14 @@ pmap_enter_ma(struct pmap *pmap, vaddr_t
*/
if ((~opte & (PG_V | PG_PVLIST)) == 0) {
- if ((old_pg = PHYS_TO_VM_PAGE(pmap_pte2pa(opte))) != NULL) {
+ if ((old_pg = PHYS_TO_VM_PAGE(oldpa)) != NULL) {
KASSERT(uvm_page_locked_p(old_pg));
old_pp = VM_PAGE_TO_PP(old_pg);
- } else if ((old_pp = pmap_pv_tracked(pmap_pte2pa(opte)))
- == NULL) {
- pa = pmap_pte2pa(opte);
- panic("pmap_enter: PG_PVLIST with pv-untracked page"
+ } else if ((old_pp = pmap_pv_tracked(oldpa)) == NULL) {
+ panic("%s: pmap_enter: PG_PVLIST with pv-untracked page"
" va = 0x%"PRIxVADDR
" pa = 0x%" PRIxPADDR " (0x%" PRIxPADDR ")",
- va, pa, atop(pa));
+ __func__, va, oldpa, atop(pa));
}
old_pve = pmap_remove_pv(old_pp, ptp, va);
@@ -4228,7 +4266,6 @@ same_pa:
error = 0;
out:
kpreempt_enable();
-out2:
if (old_pve != NULL) {
pool_cache_put(&pmap_pv_cache, old_pve);
}
Index: src/sys/dev/dtv/dtv_scatter.c
diff -u src/sys/dev/dtv/dtv_scatter.c:1.3 src/sys/dev/dtv/dtv_scatter.c:1.3.2.1
--- src/sys/dev/dtv/dtv_scatter.c:1.3 Thu Jun 1 02:45:10 2017
+++ src/sys/dev/dtv/dtv_scatter.c Tue Feb 27 09:07:33 2018
@@ -1,4 +1,4 @@
-/* $NetBSD: dtv_scatter.c,v 1.3 2017/06/01 02:45:10 chs Exp $ */
+/* $NetBSD: dtv_scatter.c,v 1.3.2.1 2018/02/27 09:07:33 martin Exp $ */
/*
* Copyright (c) 2008 Patrick Mahoney <p...@polycrystal.org>
@@ -30,7 +30,7 @@
*/
#include <sys/cdefs.h>
-__KERNEL_RCSID(0, "$NetBSD: dtv_scatter.c,v 1.3 2017/06/01 02:45:10 chs Exp $");
+__KERNEL_RCSID(0, "$NetBSD: dtv_scatter.c,v 1.3.2.1 2018/02/27 09:07:33 martin Exp $");
#include <sys/param.h>
#include <sys/ioctl.h>
@@ -101,7 +101,7 @@ dtv_scatter_buf_set_size(struct dtv_scat
sb->sb_page_ary[i] = old_ary[i];
/* allocate any new pages */
for (; i < npages; ++i) {
- sb->sb_page_ary[i] = pool_cache_get(sb->sb_pool, 0);
+ sb->sb_page_ary[i] = pool_cache_get(sb->sb_pool, PR_WAITOK);
/* TODO: does pool_cache_get return NULL on
* ENOMEM? If so, we need to release or note
* the pages with did allocate
Index: src/sys/dev/marvell/mvxpsec.c
diff -u src/sys/dev/marvell/mvxpsec.c:1.1 src/sys/dev/marvell/mvxpsec.c:1.1.12.1
--- src/sys/dev/marvell/mvxpsec.c:1.1 Wed Jun 3 04:20:02 2015
+++ src/sys/dev/marvell/mvxpsec.c Tue Feb 27 09:07:32 2018
@@ -1,4 +1,4 @@
-/* $NetBSD: mvxpsec.c,v 1.1 2015/06/03 04:20:02 hsuenaga Exp $ */
+/* $NetBSD: mvxpsec.c,v 1.1.12.1 2018/02/27 09:07:32 martin Exp $ */
/*
* Copyright (c) 2015 Internet Initiative Japan Inc.
* All rights reserved.
@@ -1498,7 +1498,7 @@ mvxpsec_session_alloc(struct mvxpsec_sof
{
struct mvxpsec_session *mv_s;
- mv_s = pool_cache_get(sc->sc_session_pool, 0);
+ mv_s = pool_cache_get(sc->sc_session_pool, PR_NOWAIT);
if (mv_s == NULL) {
log(LOG_ERR, "%s: cannot allocate memory\n", __func__);
return NULL;
@@ -1614,7 +1614,7 @@ mvxpsec_packet_alloc(struct mvxpsec_sess
sc->sc_free_qlen--;
}
else {
- mv_p = pool_cache_get(sc->sc_packet_pool, 0);
+ mv_p = pool_cache_get(sc->sc_packet_pool, PR_NOWAIT);
if (mv_p == NULL) {
log(LOG_ERR, "%s: cannot allocate memory\n",
__func__);
Index: src/sys/kern/subr_extent.c
diff -u src/sys/kern/subr_extent.c:1.80 src/sys/kern/subr_extent.c:1.80.8.1
--- src/sys/kern/subr_extent.c:1.80 Mon Dec 19 13:02:14 2016
+++ src/sys/kern/subr_extent.c Tue Feb 27 09:07:32 2018
@@ -1,4 +1,4 @@
-/* $NetBSD: subr_extent.c,v 1.80 2016/12/19 13:02:14 cherry Exp $ */
+/* $NetBSD: subr_extent.c,v 1.80.8.1 2018/02/27 09:07:32 martin Exp $ */
/*-
* Copyright (c) 1996, 1998, 2007 The NetBSD Foundation, Inc.
@@ -34,7 +34,7 @@
*/
#include <sys/cdefs.h>
-__KERNEL_RCSID(0, "$NetBSD: subr_extent.c,v 1.80 2016/12/19 13:02:14 cherry Exp $");
+__KERNEL_RCSID(0, "$NetBSD: subr_extent.c,v 1.80.8.1 2018/02/27 09:07:32 martin Exp $");
#ifdef _KERNEL
#ifdef _KERNEL_OPT
@@ -175,7 +175,7 @@ extent_alloc_region_descriptor(struct ex
}
alloc:
- rp = pool_get(&expool, (flags & EX_WAITOK) ? PR_WAITOK : 0);
+ rp = pool_get(&expool, (flags & EX_WAITOK) ? PR_WAITOK : PR_NOWAIT);
if (rp != NULL)
rp->er_flags = ER_ALLOC;
Index: src/sys/kern/subr_pool.c
diff -u src/sys/kern/subr_pool.c:1.207 src/sys/kern/subr_pool.c:1.207.6.1
--- src/sys/kern/subr_pool.c:1.207 Tue Mar 14 03:13:50 2017
+++ src/sys/kern/subr_pool.c Tue Feb 27 09:07:32 2018
@@ -1,4 +1,4 @@
-/* $NetBSD: subr_pool.c,v 1.207 2017/03/14 03:13:50 riastradh Exp $ */
+/* $NetBSD: subr_pool.c,v 1.207.6.1 2018/02/27 09:07:32 martin Exp $ */
/*-
* Copyright (c) 1997, 1999, 2000, 2002, 2007, 2008, 2010, 2014, 2015
@@ -33,7 +33,7 @@
*/
#include <sys/cdefs.h>
-__KERNEL_RCSID(0, "$NetBSD: subr_pool.c,v 1.207 2017/03/14 03:13:50 riastradh Exp $");
+__KERNEL_RCSID(0, "$NetBSD: subr_pool.c,v 1.207.6.1 2018/02/27 09:07:32 martin Exp $");
#ifdef _KERNEL_OPT
#include "opt_ddb.h"
@@ -483,7 +483,7 @@ pool_init(struct pool *pp, size_t size,
*/
TAILQ_FOREACH(pp1, &pool_head, pr_poollist) {
if (pp == pp1)
- panic("pool_init: pool %s already initialised",
+ panic("%s: [%s] already initialised", __func__,
wchan);
}
if (__predict_true(!cold))
@@ -524,8 +524,8 @@ pool_init(struct pool *pp, size_t size,
prsize = roundup(prsize, align);
KASSERTMSG((prsize <= palloc->pa_pagesz),
- "pool_init: pool item size (%zu) larger than page size (%u)",
- prsize, palloc->pa_pagesz);
+ "%s: [%s] pool item size (%zu) larger than page size (%u)",
+ __func__, wchan, prsize, palloc->pa_pagesz);
/*
* Initialize the pool structure.
@@ -608,7 +608,8 @@ pool_init(struct pool *pp, size_t size,
* if you see this panic, consider to tweak
* PHPOOL_MAX and PHPOOL_FREELIST_NELEM.
*/
- panic("%s: too large itemsperpage(%d) for PR_NOTOUCH",
+ panic("%s: [%s] too large itemsperpage(%d) for "
+ "PR_NOTOUCH", __func__,
pp->pr_wchan, pp->pr_itemsperpage);
}
pp->pr_phpool = &phpool[idx];
@@ -696,7 +697,7 @@ pool_destroy(struct pool *pp)
KASSERT(pp->pr_cache == NULL);
KASSERTMSG((pp->pr_nout == 0),
- "pool_destroy: pool busy: still out: %u", pp->pr_nout);
+ "%s: pool busy: still out: %u", __func__, pp->pr_nout);
KASSERT(LIST_EMPTY(&pp->pr_fullpages));
KASSERT(LIST_EMPTY(&pp->pr_partpages));
@@ -718,7 +719,7 @@ pool_set_drain_hook(struct pool *pp, voi
/* XXX no locking -- must be used just after pool_init() */
KASSERTMSG((pp->pr_drain_hook == NULL),
- "pool_set_drain_hook(%s): already set", pp->pr_wchan);
+ "%s: [%s] already set", __func__, pp->pr_wchan);
pp->pr_drain_hook = fn;
pp->pr_drain_hook_arg = arg;
}
@@ -729,7 +730,7 @@ pool_alloc_item_header(struct pool *pp,
struct pool_item_header *ph;
if ((pp->pr_roflags & PR_PHINPAGE) != 0)
- ph = (struct pool_item_header *) ((char *)storage + pp->pr_phoffset);
+ ph = (void *)((char *)storage + pp->pr_phoffset);
else
ph = pool_get(pp->pr_phpool, flags);
@@ -746,13 +747,14 @@ pool_get(struct pool *pp, int flags)
struct pool_item_header *ph;
void *v;
+ KASSERT(!(flags & PR_NOWAIT) != !(flags & PR_WAITOK));
KASSERTMSG((pp->pr_itemsperpage != 0),
- "pool_get: pool '%s': pr_itemsperpage is zero, "
- "pool not initialized?", pp->pr_wchan);
+ "%s: [%s] pr_itemsperpage is zero, "
+ "pool not initialized?", __func__, pp->pr_wchan);
KASSERTMSG((!(cpu_intr_p() || cpu_softintr_p())
|| pp->pr_ipl != IPL_NONE || cold || panicstr != NULL),
- "pool '%s' is IPL_NONE, but called from interrupt context",
- pp->pr_wchan);
+ "%s: [%s] is IPL_NONE, but called from interrupt context",
+ __func__, pp->pr_wchan);
if (flags & PR_WAITOK) {
ASSERT_SLEEPABLE();
}
@@ -765,7 +767,7 @@ pool_get(struct pool *pp, int flags)
* the pool.
*/
KASSERTMSG((pp->pr_nout <= pp->pr_hardlimit),
- "pool_get: %s: crossed hard limit", pp->pr_wchan);
+ "%s: %s: crossed hard limit", __func__, pp->pr_wchan);
if (__predict_false(pp->pr_nout == pp->pr_hardlimit)) {
if (pp->pr_drain_hook != NULL) {
/*
@@ -786,7 +788,9 @@ pool_get(struct pool *pp, int flags)
* it be?
*/
pp->pr_flags |= PR_WANTED;
- cv_wait(&pp->pr_cv, &pp->pr_lock);
+ do {
+ cv_wait(&pp->pr_cv, &pp->pr_lock);
+ } while (pp->pr_flags & PR_WANTED);
goto startover;
}
@@ -801,6 +805,7 @@ pool_get(struct pool *pp, int flags)
pp->pr_nfail++;
mutex_exit(&pp->pr_lock);
+ KASSERT((flags & (PR_NOWAIT|PR_LIMITFAIL)) != 0);
return (NULL);
}
@@ -814,9 +819,8 @@ pool_get(struct pool *pp, int flags)
int error;
KASSERTMSG((pp->pr_nitems == 0),
- "pool_get: nitems inconsistent"
- ": %s: curpage NULL, nitems %u",
- pp->pr_wchan, pp->pr_nitems);
+ "%s: [%s] curpage NULL, inconsistent nitems %u",
+ __func__, pp->pr_wchan, pp->pr_nitems);
/*
* Call the back-end page allocator for more memory.
@@ -826,6 +830,14 @@ pool_get(struct pool *pp, int flags)
error = pool_grow(pp, flags);
if (error != 0) {
/*
+ * pool_grow aborts when another thread
+ * is allocating a new page. Retry if it
+ * waited for it.
+ */
+ if (error == ERESTART)
+ goto startover;
+
+ /*
* We were unable to allocate a page or item
* header, but we released the lock during
* allocation, so perhaps items were freed
@@ -836,6 +848,7 @@ pool_get(struct pool *pp, int flags)
pp->pr_nfail++;
mutex_exit(&pp->pr_lock);
+ KASSERT((flags & (PR_WAITOK|PR_NOWAIT)) == PR_NOWAIT);
return (NULL);
}
@@ -844,21 +857,20 @@ pool_get(struct pool *pp, int flags)
}
if (pp->pr_roflags & PR_NOTOUCH) {
KASSERTMSG((ph->ph_nmissing < pp->pr_itemsperpage),
- "pool_get: %s: page empty", pp->pr_wchan);
+ "%s: %s: page empty", __func__, pp->pr_wchan);
v = pr_item_notouch_get(pp, ph);
} else {
v = pi = LIST_FIRST(&ph->ph_itemlist);
if (__predict_false(v == NULL)) {
mutex_exit(&pp->pr_lock);
- panic("pool_get: %s: page empty", pp->pr_wchan);
+ panic("%s: [%s] page empty", __func__, pp->pr_wchan);
}
KASSERTMSG((pp->pr_nitems > 0),
- "pool_get: nitems inconsistent"
- ": %s: items on itemlist, nitems %u",
- pp->pr_wchan, pp->pr_nitems);
+ "%s: [%s] nitems %u inconsistent on itemlist",
+ __func__, pp->pr_wchan, pp->pr_nitems);
KASSERTMSG((pi->pi_magic == PI_MAGIC),
- "pool_get(%s): free list modified: "
- "magic=%x; page %p; item addr %p",
+ "%s: [%s] free list modified: "
+ "magic=%x; page %p; item addr %p", __func__,
pp->pr_wchan, pi->pi_magic, ph->ph_page, pi);
/*
@@ -883,7 +895,8 @@ pool_get(struct pool *pp, int flags)
if (ph->ph_nmissing == pp->pr_itemsperpage) {
KASSERTMSG(((pp->pr_roflags & PR_NOTOUCH) ||
LIST_EMPTY(&ph->ph_itemlist)),
- "pool_get: %s: nmissing inconsistent", pp->pr_wchan);
+ "%s: [%s] nmissing (%u) inconsistent", __func__,
+ pp->pr_wchan, ph->ph_nmissing);
/*
* This page is now full. Move it to the full list
* and select a new current page.
@@ -929,10 +942,10 @@ pool_do_put(struct pool *pp, void *v, st
LOCKDEBUG_MEM_CHECK(v, pp->pr_size);
KASSERTMSG((pp->pr_nout > 0),
- "pool_put: pool %s: putting with none out", pp->pr_wchan);
+ "%s: [%s] putting with none out", __func__, pp->pr_wchan);
if (__predict_false((ph = pr_find_pagehead(pp, v)) == NULL)) {
- panic("pool_put: %s: page header missing", pp->pr_wchan);
+ panic("%s: [%s] page header missing", __func__, pp->pr_wchan);
}
/*
@@ -1045,26 +1058,65 @@ pool_put(struct pool *pp, void *v)
static int
pool_grow(struct pool *pp, int flags)
{
- struct pool_item_header *ph = NULL;
- char *cp;
-
- mutex_exit(&pp->pr_lock);
- cp = pool_allocator_alloc(pp, flags);
- if (__predict_true(cp != NULL)) {
- ph = pool_alloc_item_header(pp, cp, flags);
- }
- if (__predict_false(cp == NULL || ph == NULL)) {
- if (cp != NULL) {
- pool_allocator_free(pp, cp);
+ /*
+ * If there's a pool_grow in progress, wait for it to complete
+ * and try again from the top.
+ */
+ if (pp->pr_flags & PR_GROWING) {
+ if (flags & PR_WAITOK) {
+ do {
+ cv_wait(&pp->pr_cv, &pp->pr_lock);
+ } while (pp->pr_flags & PR_GROWING);
+ return ERESTART;
+ } else {
+ if (pp->pr_flags & PR_GROWINGNOWAIT) {
+ /*
+ * This needs an unlock/relock dance so
+ * that the other caller has a chance to
+ * run and actually do the thing. Note
+ * that this is effectively a busy-wait.
+ */
+ mutex_exit(&pp->pr_lock);
+ mutex_enter(&pp->pr_lock);
+ return ERESTART;
+ }
+ return EWOULDBLOCK;
}
- mutex_enter(&pp->pr_lock);
- return ENOMEM;
}
+ pp->pr_flags |= PR_GROWING;
+ if (flags & PR_WAITOK)
+ mutex_exit(&pp->pr_lock);
+ else
+ pp->pr_flags |= PR_GROWINGNOWAIT;
- mutex_enter(&pp->pr_lock);
+ char *cp = pool_allocator_alloc(pp, flags);
+ if (__predict_false(cp == NULL))
+ goto out;
+
+ struct pool_item_header *ph = pool_alloc_item_header(pp, cp, flags);
+ if (__predict_false(ph == NULL)) {
+ pool_allocator_free(pp, cp);
+ goto out;
+ }
+
+ if (flags & PR_WAITOK)
+ mutex_enter(&pp->pr_lock);
pool_prime_page(pp, cp, ph);
pp->pr_npagealloc++;
+ KASSERT(pp->pr_flags & PR_GROWING);
+ pp->pr_flags &= ~(PR_GROWING|PR_GROWINGNOWAIT);
+ /*
+ * If anyone was waiting for pool_grow, notify them that we
+ * may have just done it.
+ */
+ cv_broadcast(&pp->pr_cv);
return 0;
+out:
+ if (flags & PR_WAITOK)
+ mutex_enter(&pp->pr_lock);
+ KASSERT(pp->pr_flags & PR_GROWING);
+ pp->pr_flags &= ~(PR_GROWING|PR_GROWINGNOWAIT);
+ return ENOMEM;
}
/*
@@ -1080,12 +1132,15 @@ pool_prime(struct pool *pp, int n)
newpages = roundup(n, pp->pr_itemsperpage) / pp->pr_itemsperpage;
- while (newpages-- > 0) {
+ while (newpages > 0) {
error = pool_grow(pp, PR_NOWAIT);
if (error) {
+ if (error == ERESTART)
+ continue;
break;
}
pp->pr_minpages++;
+ newpages--;
}
if (pp->pr_minpages >= pp->pr_maxpages)
@@ -1112,7 +1167,7 @@ pool_prime_page(struct pool *pp, void *s
KASSERT(mutex_owned(&pp->pr_lock));
KASSERTMSG(((pp->pr_roflags & PR_NOALIGN) ||
(((uintptr_t)cp & (pp->pr_alloc->pa_pagesz - 1)) == 0)),
- "pool_prime_page: %s: unaligned page: %p", pp->pr_wchan, cp);
+ "%s: [%s] unaligned page: %p", __func__, pp->pr_wchan, cp);
/*
* Insert page header.
@@ -1195,6 +1250,8 @@ pool_catchup(struct pool *pp)
while (POOL_NEEDS_CATCHUP(pp)) {
error = pool_grow(pp, PR_NOWAIT);
if (error) {
+ if (error == ERESTART)
+ continue;
break;
}
}
@@ -2134,8 +2191,10 @@ pool_cache_get_slow(pool_cache_cpu_t *cc
object = pool_get(&pc->pc_pool, flags);
*objectp = object;
- if (__predict_false(object == NULL))
+ if (__predict_false(object == NULL)) {
+ KASSERT((flags & (PR_WAITOK|PR_NOWAIT)) == PR_NOWAIT);
return false;
+ }
if (__predict_false((*pc->pc_ctor)(pc->pc_arg, object, flags) != 0)) {
pool_put(&pc->pc_pool, object);
@@ -2173,10 +2232,11 @@ pool_cache_get_paddr(pool_cache_t pc, in
void *object;
int s;
+ KASSERT(!(flags & PR_NOWAIT) != !(flags & PR_WAITOK));
KASSERTMSG((!cpu_intr_p() && !cpu_softintr_p()) ||
(pc->pc_pool.pr_ipl != IPL_NONE || cold || panicstr != NULL),
- "pool '%s' is IPL_NONE, but called from interrupt context\n",
- pc->pc_pool.pr_wchan);
+ "%s: [%s] is IPL_NONE, but called from interrupt context",
+ __func__, pc->pc_pool.pr_wchan);
if (flags & PR_WAITOK) {
ASSERT_SLEEPABLE();
@@ -2226,6 +2286,11 @@ pool_cache_get_paddr(pool_cache_t pc, in
break;
}
+ /*
+ * We would like to KASSERT(object || (flags & PR_NOWAIT)), but
+ * pool_cache_get can fail even in the PR_WAITOK case, if the
+ * constructor fails.
+ */
return object;
}
Index: src/sys/kern/uipc_mbuf.c
diff -u src/sys/kern/uipc_mbuf.c:1.172 src/sys/kern/uipc_mbuf.c:1.172.6.1
--- src/sys/kern/uipc_mbuf.c:1.172 Fri Mar 31 05:44:05 2017
+++ src/sys/kern/uipc_mbuf.c Tue Feb 27 09:07:32 2018
@@ -1,4 +1,4 @@
-/* $NetBSD: uipc_mbuf.c,v 1.172 2017/03/31 05:44:05 msaitoh Exp $ */
+/* $NetBSD: uipc_mbuf.c,v 1.172.6.1 2018/02/27 09:07:32 martin Exp $ */
/*-
* Copyright (c) 1999, 2001 The NetBSD Foundation, Inc.
@@ -62,7 +62,7 @@
*/
#include <sys/cdefs.h>
-__KERNEL_RCSID(0, "$NetBSD: uipc_mbuf.c,v 1.172 2017/03/31 05:44:05 msaitoh Exp $");
+__KERNEL_RCSID(0, "$NetBSD: uipc_mbuf.c,v 1.172.6.1 2018/02/27 09:07:32 martin Exp $");
#ifdef _KERNEL_OPT
#include "opt_mbuftrace.h"
@@ -589,7 +589,7 @@ m_get(int nowait, int type)
KASSERT(type != MT_FREE);
m = pool_cache_get(mb_cache,
- nowait == M_WAIT ? PR_WAITOK|PR_LIMITFAIL : 0);
+ nowait == M_WAIT ? PR_WAITOK|PR_LIMITFAIL : PR_NOWAIT);
if (m == NULL)
return NULL;
Index: src/sys/opencrypto/crypto.c
diff -u src/sys/opencrypto/crypto.c:1.78.2.4 src/sys/opencrypto/crypto.c:1.78.2.5
--- src/sys/opencrypto/crypto.c:1.78.2.4 Tue Jan 2 10:38:36 2018
+++ src/sys/opencrypto/crypto.c Tue Feb 27 09:07:32 2018
@@ -1,4 +1,4 @@
-/* $NetBSD: crypto.c,v 1.78.2.4 2018/01/02 10:38:36 snj Exp $ */
+/* $NetBSD: crypto.c,v 1.78.2.5 2018/02/27 09:07:32 martin Exp $ */
/* $FreeBSD: src/sys/opencrypto/crypto.c,v 1.4.2.5 2003/02/26 00:14:05 sam Exp $ */
/* $OpenBSD: crypto.c,v 1.41 2002/07/17 23:52:38 art Exp $ */
@@ -53,7 +53,7 @@
*/
#include <sys/cdefs.h>
-__KERNEL_RCSID(0, "$NetBSD: crypto.c,v 1.78.2.4 2018/01/02 10:38:36 snj Exp $");
+__KERNEL_RCSID(0, "$NetBSD: crypto.c,v 1.78.2.5 2018/02/27 09:07:32 martin Exp $");
#include <sys/param.h>
#include <sys/reboot.h>
@@ -1655,14 +1655,14 @@ crypto_getreq(int num)
}
crypto_put_crp_ret_qs(curcpu());
- crp = pool_cache_get(cryptop_cache, 0);
+ crp = pool_cache_get(cryptop_cache, PR_NOWAIT);
if (crp == NULL) {
return NULL;
}
memset(crp, 0, sizeof(struct cryptop));
while (num--) {
- crd = pool_cache_get(cryptodesc_cache, 0);
+ crd = pool_cache_get(cryptodesc_cache, PR_NOWAIT);
if (crd == NULL) {
crypto_freereq(crp);
return NULL;
Index: src/sys/sys/mbuf.h
diff -u src/sys/sys/mbuf.h:1.170.2.1 src/sys/sys/mbuf.h:1.170.2.2
--- src/sys/sys/mbuf.h:1.170.2.1 Tue Oct 24 08:39:00 2017
+++ src/sys/sys/mbuf.h Tue Feb 27 09:07:33 2018
@@ -1,4 +1,4 @@
-/* $NetBSD: mbuf.h,v 1.170.2.1 2017/10/24 08:39:00 snj Exp $ */
+/* $NetBSD: mbuf.h,v 1.170.2.2 2018/02/27 09:07:33 martin Exp $ */
/*-
* Copyright (c) 1996, 1997, 1999, 2001, 2007 The NetBSD Foundation, Inc.
@@ -516,7 +516,7 @@ do { \
do { \
(m)->m_ext_storage.ext_buf = (char *) \
pool_cache_get_paddr((pool_cache), \
- (how) == M_WAIT ? (PR_WAITOK|PR_LIMITFAIL) : 0, \
+ (how) == M_WAIT ? (PR_WAITOK|PR_LIMITFAIL) : PR_NOWAIT, \
&(m)->m_ext_storage.ext_paddr); \
if ((m)->m_ext_storage.ext_buf != NULL) { \
MCLINITREFERENCE(m); \
Index: src/sys/sys/pool.h
diff -u src/sys/sys/pool.h:1.79 src/sys/sys/pool.h:1.79.10.1
--- src/sys/sys/pool.h:1.79 Wed Jul 29 00:10:25 2015
+++ src/sys/sys/pool.h Tue Feb 27 09:07:33 2018
@@ -1,4 +1,4 @@
-/* $NetBSD: pool.h,v 1.79 2015/07/29 00:10:25 christos Exp $ */
+/* $NetBSD: pool.h,v 1.79.10.1 2018/02/27 09:07:33 martin Exp $ */
/*-
* Copyright (c) 1997, 1998, 1999, 2000, 2007 The NetBSD Foundation, Inc.
@@ -147,6 +147,8 @@ struct pool {
#define PR_NOTOUCH 0x400 /* don't use free items to keep internal state*/
#define PR_NOALIGN 0x800 /* don't assume backend alignment */
#define PR_LARGECACHE 0x1000 /* use large cache groups */
+#define PR_GROWING 0x2000 /* pool_grow in progress */
+#define PR_GROWINGNOWAIT 0x4000 /* pool_grow in progress by PR_NOWAIT alloc */
/*
* `pr_lock' protects the pool's data structures when removing
Index: src/sys/ufs/chfs/chfs_malloc.c
diff -u src/sys/ufs/chfs/chfs_malloc.c:1.4 src/sys/ufs/chfs/chfs_malloc.c:1.4.30.1
--- src/sys/ufs/chfs/chfs_malloc.c:1.4 Fri Oct 19 12:44:39 2012
+++ src/sys/ufs/chfs/chfs_malloc.c Tue Feb 27 09:07:33 2018
@@ -1,4 +1,4 @@
-/* $NetBSD: chfs_malloc.c,v 1.4 2012/10/19 12:44:39 ttoth Exp $ */
+/* $NetBSD: chfs_malloc.c,v 1.4.30.1 2018/02/27 09:07:33 martin Exp $ */
/*-
* Copyright (c) 2010 Department of Software Engineering,
@@ -314,7 +314,7 @@ struct chfs_flash_vnode*
chfs_alloc_flash_vnode(void)
{
struct chfs_flash_vnode *ret;
- ret = pool_cache_get(chfs_flash_vnode_cache, 0);
+ ret = pool_cache_get(chfs_flash_vnode_cache, PR_WAITOK);
return ret;
}
@@ -330,7 +330,7 @@ struct chfs_flash_dirent_node*
chfs_alloc_flash_dirent(void)
{
struct chfs_flash_dirent_node *ret;
- ret = pool_cache_get(chfs_flash_dirent_cache, 0);
+ ret = pool_cache_get(chfs_flash_dirent_cache, PR_WAITOK);
return ret;
}
@@ -346,7 +346,7 @@ struct chfs_flash_data_node*
chfs_alloc_flash_dnode(void)
{
struct chfs_flash_data_node *ret;
- ret = pool_cache_get(chfs_flash_dnode_cache, 0);
+ ret = pool_cache_get(chfs_flash_dnode_cache, PR_WAITOK);
return ret;
}
@@ -362,7 +362,7 @@ struct chfs_node_frag*
chfs_alloc_node_frag(void)
{
struct chfs_node_frag *ret;
- ret = pool_cache_get(chfs_node_frag_cache, 0);
+ ret = pool_cache_get(chfs_node_frag_cache, PR_WAITOK);
return ret;
}
@@ -378,7 +378,7 @@ struct chfs_tmp_dnode *
chfs_alloc_tmp_dnode(void)
{
struct chfs_tmp_dnode *ret;
- ret = pool_cache_get(chfs_tmp_dnode_cache, 0);
+ ret = pool_cache_get(chfs_tmp_dnode_cache, PR_WAITOK);
ret->next = NULL;
return ret;
}
@@ -395,7 +395,7 @@ struct chfs_tmp_dnode_info *
chfs_alloc_tmp_dnode_info(void)
{
struct chfs_tmp_dnode_info *ret;
- ret = pool_cache_get(chfs_tmp_dnode_info_cache, 0);
+ ret = pool_cache_get(chfs_tmp_dnode_info_cache, PR_WAITOK);
ret->tmpnode = NULL;
return ret;
}
Index: src/sys/uvm/uvm_fault.c
diff -u src/sys/uvm/uvm_fault.c:1.199.6.2 src/sys/uvm/uvm_fault.c:1.199.6.3
--- src/sys/uvm/uvm_fault.c:1.199.6.2 Thu Nov 2 21:29:53 2017
+++ src/sys/uvm/uvm_fault.c Tue Feb 27 09:07:33 2018
@@ -1,4 +1,4 @@
-/* $NetBSD: uvm_fault.c,v 1.199.6.2 2017/11/02 21:29:53 snj Exp $ */
+/* $NetBSD: uvm_fault.c,v 1.199.6.3 2018/02/27 09:07:33 martin Exp $ */
/*
* Copyright (c) 1997 Charles D. Cranor and Washington University.
@@ -32,7 +32,7 @@
*/
#include <sys/cdefs.h>
-__KERNEL_RCSID(0, "$NetBSD: uvm_fault.c,v 1.199.6.2 2017/11/02 21:29:53 snj Exp $");
+__KERNEL_RCSID(0, "$NetBSD: uvm_fault.c,v 1.199.6.3 2018/02/27 09:07:33 martin Exp $");
#include "opt_uvmhist.h"
@@ -1211,7 +1211,7 @@ uvm_fault_upper_lookup(
}
/*
- * uvm_fault_upper_neighbor: enter single lower neighbor page.
+ * uvm_fault_upper_neighbor: enter single upper neighbor page.
*
* => called with amap and anon locked.
*/
@@ -1493,6 +1493,8 @@ uvm_fault_upper_enter(
struct uvm_object *uobj, struct vm_anon *anon, struct vm_page *pg,
struct vm_anon *oanon)
{
+ struct pmap *pmap = ufi->orig_map->pmap;
+ vaddr_t va = ufi->orig_rvaddr;
struct vm_amap * const amap = ufi->entry->aref.ar_amap;
UVMHIST_FUNC("uvm_fault_upper_enter"); UVMHIST_CALLED(maphist);
@@ -1508,14 +1510,26 @@ uvm_fault_upper_enter(
UVMHIST_LOG(maphist,
" MAPPING: anon: pm=%#jx, va=%#jx, pg=%#jx, promote=%jd",
- (uintptr_t)ufi->orig_map->pmap, ufi->orig_rvaddr,
- (uintptr_t)pg, flt->promote);
- if (pmap_enter(ufi->orig_map->pmap, ufi->orig_rvaddr,
- VM_PAGE_TO_PHYS(pg),
+ (uintptr_t)pmap, va, (uintptr_t)pg, flt->promote);
+ if (pmap_enter(pmap, va, VM_PAGE_TO_PHYS(pg),
flt->enter_prot, flt->access_type | PMAP_CANFAIL |
(flt->wire_mapping ? PMAP_WIRED : 0)) != 0) {
/*
+ * If pmap_enter() fails, it must not leave behind an existing
+ * pmap entry. In particular, a now-stale entry for a different
+ * page would leave the pmap inconsistent with the vm_map.
+ * This is not to imply that pmap_enter() should remove an
+ * existing mapping in such a situation (since that could create
+ * different problems, eg. if the existing mapping is wired),
+ * but rather that the pmap should be designed such that it
+ * never needs to fail when the new mapping is replacing an
+ * existing mapping and the new page has no existing mappings.
+ */
+
+ KASSERT(!pmap_extract(pmap, va, NULL));
+
+ /*
* No need to undo what we did; we can simply think of
* this as the pmap throwing away the mapping information.
*
@@ -1541,7 +1555,7 @@ uvm_fault_upper_enter(
* done case 1! finish up by unlocking everything and returning success
*/
- pmap_update(ufi->orig_map->pmap);
+ pmap_update(pmap);
uvmfault_unlockall(ufi, amap, uobj);
return 0;
}
