svn commit: r367281 - in head/sys: amd64/amd64 arm64/arm64 mips/mips powerpc/powerpc riscv/riscv
Author: alc
Date: Mon Nov 2 19:20:06 2020
New Revision: 367281
URL: https://svnweb.freebsd.org/changeset/base/367281
Log:
Tidy up the #includes. Recent changes, such as the introduction of
VM_ALLOC_WAITOK and vm_page_unwire_noq(), have eliminated the need for
many of the #includes.
Reviewed by: kib, markj
Differential Revision:https://reviews.freebsd.org/D27052
Modified:
head/sys/amd64/amd64/uma_machdep.c
head/sys/arm64/arm64/uma_machdep.c
head/sys/mips/mips/uma_machdep.c
head/sys/powerpc/powerpc/uma_machdep.c
head/sys/riscv/riscv/uma_machdep.c
Modified: head/sys/amd64/amd64/uma_machdep.c
==
--- head/sys/amd64/amd64/uma_machdep.c Mon Nov 2 18:48:06 2020
(r367280)
+++ head/sys/amd64/amd64/uma_machdep.c Mon Nov 2 19:20:06 2020
(r367281)
@@ -30,15 +30,10 @@
__FBSDID("$FreeBSD$");
#include
-#include
#include
-#include
-#include
-#include
#include
#include
#include
-#include
#include
#include
#include
Modified: head/sys/arm64/arm64/uma_machdep.c
==
--- head/sys/arm64/arm64/uma_machdep.c Mon Nov 2 18:48:06 2020
(r367280)
+++ head/sys/arm64/arm64/uma_machdep.c Mon Nov 2 19:20:06 2020
(r367281)
@@ -28,11 +28,7 @@
__FBSDID("$FreeBSD$");
#include
-#include
#include
-#include
-#include
-#include
#include
#include
#include
Modified: head/sys/mips/mips/uma_machdep.c
==
--- head/sys/mips/mips/uma_machdep.cMon Nov 2 18:48:06 2020
(r367280)
+++ head/sys/mips/mips/uma_machdep.cMon Nov 2 19:20:06 2020
(r367281)
@@ -30,11 +30,7 @@
__FBSDID("$FreeBSD$");
#include
-#include
#include
-#include
-#include
-#include
#include
#include
#include
@@ -42,7 +38,6 @@ __FBSDID("$FreeBSD$");
#include
#include
#include
-#include
void *
uma_small_alloc(uma_zone_t zone, vm_size_t bytes, int domain, u_int8_t *flags,
Modified: head/sys/powerpc/powerpc/uma_machdep.c
==
--- head/sys/powerpc/powerpc/uma_machdep.c Mon Nov 2 18:48:06 2020
(r367280)
+++ head/sys/powerpc/powerpc/uma_machdep.c Mon Nov 2 19:20:06 2020
(r367281)
@@ -28,21 +28,14 @@
__FBSDID("$FreeBSD$");
#include
-#include
-#include
-#include
-#include
#include
+#include
#include
-#include
#include
#include
+#include
#include
-#include
-#include
-#include
#include
-#include
#include
#include
#include
Modified: head/sys/riscv/riscv/uma_machdep.c
==
--- head/sys/riscv/riscv/uma_machdep.c Mon Nov 2 18:48:06 2020
(r367280)
+++ head/sys/riscv/riscv/uma_machdep.c Mon Nov 2 19:20:06 2020
(r367281)
@@ -28,19 +28,13 @@
__FBSDID("$FreeBSD$");
#include
-#include
#include
-#include
-#include
-#include
#include
#include
#include
-#include
#include
#include
#include
-#include
void *
uma_small_alloc(uma_zone_t zone, vm_size_t bytes, int domain, u_int8_t *flags,
___
svn-src-all@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-all
To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"
svn commit: r367087 - head/lib/libc/sys
Author: alc Date: Tue Oct 27 18:08:33 2020 New Revision: 367087 URL: https://svnweb.freebsd.org/changeset/base/367087 Log: Revise the description of MAP_STACK. In particular, describe the guard in more detail. Reviewed by: bcr, kib, markj MFC after:1 week Differential Revision:https://reviews.freebsd.org/D26908 Modified: head/lib/libc/sys/mmap.2 Modified: head/lib/libc/sys/mmap.2 == --- head/lib/libc/sys/mmap.2Tue Oct 27 17:25:40 2020(r367086) +++ head/lib/libc/sys/mmap.2Tue Oct 27 18:08:33 2020(r367087) @@ -28,7 +28,7 @@ .\"@(#)mmap.2 8.4 (Berkeley) 5/11/95 .\" $FreeBSD$ .\" -.Dd October 21, 2020 +.Dd October 27, 2020 .Dt MMAP 2 .Os .Sh NAME @@ -319,9 +319,30 @@ Modifications are private. .It Dv MAP_SHARED Modifications are shared. .It Dv MAP_STACK +Creates both a mapped region that grows downward on demand and an +adjoining guard that both reserves address space for the mapped region +to grow into and limits the mapped region's growth. +Together, the mapped region and the guard occupy +.Fa len +bytes of the address space. +The guard starts at the returned address, and the mapped region ends at +the returned address plus +.Fa len +bytes. +Upon access to the guard, the mapped region automatically grows in size, +and the guard shrinks by an equal amount. +Essentially, the boundary between the guard and the mapped region moves +downward so that the access falls within the enlarged mapped region. +However, the guard will never shrink to less than the number of pages +specified by the sysctl +.Dv security.bsd.stack_guard_page , +thereby ensuring that a gap for detecting stack overflow always exists +between the downward growing mapped region and the closest mapped region +beneath it. +.Pp .Dv MAP_STACK implies -.Dv MAP_ANON , +.Dv MAP_ANON and .Fa offset of 0. @@ -334,23 +355,6 @@ must include at least .Dv PROT_READ and .Dv PROT_WRITE . -.Pp -This option creates -a memory region that grows to at most -.Fa len -bytes in size, starting from the stack top and growing down. -The -stack top is the starting address returned by the call, plus -.Fa len -bytes. -The bottom of the stack at maximum growth is the starting -address returned by the call. -.Pp -Stacks created with -.Dv MAP_STACK -automatically grow. -Guards prevent inadvertent use of the regions into which those -stacks can grow without requiring mapping the whole stack in advance. The size of the guard, in pages, is specified by sysctl .Dv security.bsd.stack_guard_page . .El ___ svn-src-all@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"
svn commit: r366960 - head/sys/vm
Author: alc
Date: Fri Oct 23 06:24:38 2020
New Revision: 366960
URL: https://svnweb.freebsd.org/changeset/base/366960
Log:
Conditionally compile struct vm_phys_seg's md_first field. This field is
only used by arm64's pmap.
Reviewed by: kib, markj, scottph
Differential Revision:https://reviews.freebsd.org/D26907
Modified:
head/sys/vm/vm_phys.h
Modified: head/sys/vm/vm_phys.h
==
--- head/sys/vm/vm_phys.h Fri Oct 23 02:24:43 2020(r366959)
+++ head/sys/vm/vm_phys.h Fri Oct 23 06:24:38 2020(r366960)
@@ -71,7 +71,9 @@ struct vm_phys_seg {
#if VM_NRESERVLEVEL > 0
vm_reserv_t first_reserv;
#endif
+#ifdef __aarch64__
void*md_first;
+#endif
int domain;
struct vm_freelist (*free_queues)[VM_NFREEPOOL][VM_NFREEORDER_MAX];
};
___
svn-src-all@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-all
To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"
svn commit: r366944 - head/sys/arm64/arm64
Author: alc
Date: Thu Oct 22 17:47:51 2020
New Revision: 366944
URL: https://svnweb.freebsd.org/changeset/base/366944
Log:
Micro-optimize uma_small_alloc(). Replace bzero(..., PAGE_SIZE) by
pagezero(). Ultimately, they use the same method for bulk zeroing, but
the generality of bzero() requires size and alignment checks that
pagezero() does not.
Eliminate an unnecessary #include.
Reviewed by: emaste, markj
MFC after:1 week
Differential Revision:https://reviews.freebsd.org/D26876
Modified:
head/sys/arm64/arm64/uma_machdep.c
Modified: head/sys/arm64/arm64/uma_machdep.c
==
--- head/sys/arm64/arm64/uma_machdep.c Thu Oct 22 17:46:55 2020
(r366943)
+++ head/sys/arm64/arm64/uma_machdep.c Thu Oct 22 17:47:51 2020
(r366944)
@@ -36,11 +36,10 @@ __FBSDID("$FreeBSD$");
#include
#include
#include
-#include
#include
#include
#include
-#include
+#include
void *
uma_small_alloc(uma_zone_t zone, vm_size_t bytes, int domain, u_int8_t *flags,
@@ -60,7 +59,7 @@ uma_small_alloc(uma_zone_t zone, vm_size_t bytes, int
dump_add_page(pa);
va = (void *)PHYS_TO_DMAP(pa);
if ((wait & M_ZERO) && (m->flags & PG_ZERO) == 0)
- bzero(va, PAGE_SIZE);
+ pagezero(va);
return (va);
}
___
svn-src-all@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-all
To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"
Re: svn commit: r363842 - head/sys/compat/linuxkpi/common/include/linux
On 8/4/20 10:25 AM, Emmanuel Vadot wrote:
Author: manu
Date: Tue Aug 4 15:25:22 2020
New Revision: 363842
URL: https://svnweb.freebsd.org/changeset/base/363842
Log:
linuxkpi: Add clear_bit_unlock
This calls clear_bit and adds a memory barrier.
Sponsored by: The FreeBSD Foundation
Reviewed by: hselasky
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D25943
Modified:
head/sys/compat/linuxkpi/common/include/linux/bitops.h
Modified: head/sys/compat/linuxkpi/common/include/linux/bitops.h
==
--- head/sys/compat/linuxkpi/common/include/linux/bitops.h Tue Aug 4
15:00:02 2020(r363841)
+++ head/sys/compat/linuxkpi/common/include/linux/bitops.h Tue Aug 4
15:25:22 2020(r363842)
@@ -275,6 +275,13 @@ find_next_zero_bit(const unsigned long *addr, unsigned
#define test_bit(i, a)
\
!!(READ_ONCE(((volatile const unsigned long *)(a))[BIT_WORD(i)]) &
BIT_MASK(i))
+static inline void
+clear_bit_unlock(long bit, volatile unsigned long *var)
+{
+ clear_bit(bit, var);
+ wmb();
For an unlock operation, the memory barrier should come before the
clear_bit() call, not after. See, for example, the alpha implementation
in Linux. Also, the correct "spelling" for this memory barrier in
FreeBSD would be atomic_thread_fence_rel(). See, for example, the
comment at the top of sys/amd64/include/atomic.h.
+}
+
static inline int
test_and_clear_bit(long bit, volatile unsigned long *var)
{
___
svn-src-all@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-all
To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"
Re: svn commit: r358733 - head/sys/sys
On 3/11/20 9:16 PM, Mateusz Guzik wrote:
On 3/10/20, Konstantin Belousov wrote:
On Tue, Mar 10, 2020 at 12:22:09AM +0100, Mateusz Guzik wrote:
On 3/9/20, Konstantin Belousov wrote:
On Mon, Mar 09, 2020 at 01:56:17AM +0100, Mateusz Guzik wrote:
On 3/8/20, Mateusz Guzik wrote:
Author: mjg
Date: Sun Mar 8 00:22:32 2020
New Revision: 358733
URL: https://svnweb.freebsd.org/changeset/base/358733
Log:
seqc: tidy up
- avoid hand-rolled read
- match begin/end in terms of fence style
There were off lists questions about this so let me clarify.
The first bit is a cosmetic change, but the second one is not.
Modified:
head/sys/sys/seqc.h
Modified: head/sys/sys/seqc.h
==
--- head/sys/sys/seqc.h Sat Mar 7 15:37:23 2020(r358732)
+++ head/sys/sys/seqc.h Sun Mar 8 00:22:32 2020(r358733)
@@ -66,7 +66,8 @@ static __inline void
seqc_write_end(seqc_t *seqcp)
{
- atomic_store_rel_int(seqcp, *seqcp + 1);
+ atomic_thread_fence_rel();
+ *seqcp += 1;
MPASS(!seqc_in_modify(*seqcp));
critical_exit();
}
For correct operation the counter has to be modified *before* any work
is done and *after* it is completed.
Consider a trivial case:
seqc++;
foo[0] = 0;
foo[1] = 1;
seqc++;
There are 2 ways in which this can be mucked with:
- the compiler can be looking at reordering or reworking the increment
(e.g., to collapse it to just += 2 instead). a compiler barrier
prevents that.
- even with generated machine code which increments in correct places
the cpu can be looking at reordering the operation (which is heavily
dependent on architecture), in particular it could end up issuing
seqc++; foo[1] = 1; which would defeat the point. This is where
release fences come in.
With the patched code there is:
seqc++;
atomic_thread_fence_rel(); /* make sure seqc++ is issued before foo
starts getting modified */
foo[0] = 0;
foo[1] = 1;
atomic_thread_fence_rel(); /* make sure modifications to foo don't
leak past this spot */
seqc++;
In comparison, the previous code was:
seqc++;
atomic_thread_fence_rel(); /* make sure seqc++ is issued before foo
starts getting modified */
foo[0] = 0;
foo[1] = 1;
atomic_store_rel_int(seqcp, *seqcp + 1); /* make sure modifications to
too don't leak past this spot and update seqc immediately */
There are 2 differences here -- one is an improvement and second one
does not matter:
the win is: the previous code forces the compiler to compute an
incremented value and then store it. On amd64 this translates to the
following (with rdx holding the address of the counter):
mov(%rdx),%eax /* load the value */
add$0x1,%eax/* add 1 */
mov%eax,(%rdx) /* store it */
On patched kernel this is:
addl $0x1,(%rdx)
which is clearly much nicer.
I am not sure that the new code on amd64 is much nicer.
But the point was that on non-amd64, i.e. armv8 and potentially on
powerpc 3.0, the patch substitutes release store with full barrier. The
later is (much) slower.
If an arch performs something significantly more expensive here it's
probably a performance bug in the port.
It is a question of how much hardware support for fine-grained fences.
Relaxed architectures add some optimized operations already, but did not
(yet) implemented complete C11 optimized set.
I think they eventually converge.
Until they did not, I think it is more important to not pessimize
to-be Tier1 arches than to collapse three fast integer instructions
on amd64 into one rw op.
But perhaps more importantly
there are significantly more frequent users of
atomic_thread_fence_rel, like refcount(9). If the issue is real that
should be looked at.
For refcount(9) use of release fence removal, we would need
atomic_fetchadd_rel_int().
To reiterate, if atomic_thread_fence_rel is indeed very expensive on
certain architectures compared to other _rel variants, the kernel is
already heavily pessimized because of use of said standalone fence in
refcount(9) and few other places and this needs to be fixed.
Just to be clear, this is not just a matter of whether our release
fences are poorly or inefficiently implemented. Release fences (as
defined by the C/C++ standards that inspire our atomics) have
*different* semantics than release stores. Specifically, a release fence
places more ordering restrictions on the nearby store accesses than a
release store does. (In fact, "man 9 atomic" describes this.) Thus, it
is almost inevitable that a release fence will have a higher cost than
the best possible implementation of a release store.
If the cost can't be reduced (e.g., perhaps this can do a no-op store
+ fence somewhere akin to how lock xadd 0,(%esp) trickery used to
work), then the atomic APIs do indeed not to be completed w.r.t.
adding all the missing acq/rel variants (that's at least fetchadd and
swap).
We can also extend the API to do relevant addition/subtraction without
svn commit: r356354 - in head/sys: amd64/amd64 arm64/arm64 i386/i386
Author: alc
Date: Sat Jan 4 19:50:25 2020
New Revision: 356354
URL: https://svnweb.freebsd.org/changeset/base/356354
Log:
When a copy-on-write fault occurs, pmap_enter() is called on to replace the
mapping to the old read-only page with a mapping to the new read-write page.
To destroy the old mapping, pmap_enter() must destroy its page table and PV
entries and invalidate its TLB entry. This change simply invalidates that
TLB entry a little earlier, specifically, on amd64 and arm64, before the PV
list lock is held.
Reviewed by: kib, markj
MFC after:1 week
Differential Revision:https://reviews.freebsd.org/D23027
Modified:
head/sys/amd64/amd64/pmap.c
head/sys/arm64/arm64/pmap.c
head/sys/i386/i386/pmap.c
Modified: head/sys/amd64/amd64/pmap.c
==
--- head/sys/amd64/amd64/pmap.c Sat Jan 4 19:29:25 2020(r356353)
+++ head/sys/amd64/amd64/pmap.c Sat Jan 4 19:50:25 2020(r356354)
@@ -6131,8 +6131,10 @@ retry:
*/
if ((origpte & (PG_M | PG_RW)) == (PG_M | PG_RW))
vm_page_dirty(om);
- if ((origpte & PG_A) != 0)
+ if ((origpte & PG_A) != 0) {
+ pmap_invalidate_page(pmap, va);
vm_page_aflag_set(om, PGA_REFERENCED);
+ }
CHANGE_PV_LIST_LOCK_TO_PHYS(, opa);
pv = pmap_pvh_remove(>md, pmap, va);
KASSERT(pv != NULL,
@@ -6144,9 +6146,13 @@ retry:
((om->flags & PG_FICTITIOUS) != 0 ||
TAILQ_EMPTY(_to_pvh(opa)->pv_list)))
vm_page_aflag_clear(om, PGA_WRITEABLE);
- }
- if ((origpte & PG_A) != 0)
+ } else {
+ /*
+* Since this mapping is unmanaged, assume that PG_A
+* is set.
+*/
pmap_invalidate_page(pmap, va);
+ }
origpte = 0;
} else {
/*
Modified: head/sys/arm64/arm64/pmap.c
==
--- head/sys/arm64/arm64/pmap.c Sat Jan 4 19:29:25 2020(r356353)
+++ head/sys/arm64/arm64/pmap.c Sat Jan 4 19:50:25 2020(r356354)
@@ -3449,8 +3449,10 @@ havel3:
*/
if (pmap_pte_dirty(orig_l3))
vm_page_dirty(om);
- if ((orig_l3 & ATTR_AF) != 0)
+ if ((orig_l3 & ATTR_AF) != 0) {
+ pmap_invalidate_page(pmap, va);
vm_page_aflag_set(om, PGA_REFERENCED);
+ }
CHANGE_PV_LIST_LOCK_TO_PHYS(, opa);
pv = pmap_pvh_remove(>md, pmap, va);
if ((m->oflags & VPO_UNMANAGED) != 0)
@@ -3460,8 +3462,11 @@ havel3:
((om->flags & PG_FICTITIOUS) != 0 ||
TAILQ_EMPTY(_to_pvh(opa)->pv_list)))
vm_page_aflag_clear(om, PGA_WRITEABLE);
+ } else {
+ KASSERT((orig_l3 & ATTR_AF) != 0,
+ ("pmap_enter: unmanaged mapping lacks ATTR_AF"));
+ pmap_invalidate_page(pmap, va);
}
- pmap_invalidate_page(pmap, va);
orig_l3 = 0;
} else {
/*
Modified: head/sys/i386/i386/pmap.c
==
--- head/sys/i386/i386/pmap.c Sat Jan 4 19:29:25 2020(r356353)
+++ head/sys/i386/i386/pmap.c Sat Jan 4 19:50:25 2020(r356354)
@@ -3798,8 +3798,10 @@ __CONCAT(PMTYPE, enter)(pmap_t pmap, vm_offset_t va, v
*/
if ((origpte & (PG_M | PG_RW)) == (PG_M | PG_RW))
vm_page_dirty(om);
- if ((origpte & PG_A) != 0)
+ if ((origpte & PG_A) != 0) {
+ pmap_invalidate_page_int(pmap, va);
vm_page_aflag_set(om, PGA_REFERENCED);
+ }
pv = pmap_pvh_remove(>md, pmap, va);
KASSERT(pv != NULL,
("pmap_enter: no PV entry for %#x", va));
@@ -3810,9 +3812,13 @@ __CONCAT(PMTYPE, enter)(pmap_t pmap, vm_offset_t va, v
((om->flags & PG_FICTITIOUS) != 0 ||
TAILQ_EMPTY(_to_pvh(opa)->pv_list)))
vm_page_aflag_clear(om, PGA_WRITEABLE);
- }
- if ((origpte & PG_A) !=
Re: svn commit: r356207 - in head/sys/arm64: arm64 include
On 12/30/19 2:59 PM, Colin Percival wrote: On 2019-12-30 12:30, Alan Cox wrote: Author: alc Date: Mon Dec 30 20:30:31 2019 New Revision: 356207 URL: https://svnweb.freebsd.org/changeset/base/356207 Log: Determine whether the MMU hardware is capable of updating a page table entry's access flag and dirty state, and enable this feature when it's available. Am I right in thinking that this should result in improved performance under load? (I'm wondering if it's time to redo my amd64 vs arm64 EC2 benchmarks.) Yes, a modest one. ___ svn-src-all@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"
svn commit: r356207 - in head/sys/arm64: arm64 include
Author: alc
Date: Mon Dec 30 20:30:31 2019
New Revision: 356207
URL: https://svnweb.freebsd.org/changeset/base/356207
Log:
Determine whether the MMU hardware is capable of updating a page table
entry's access flag and dirty state, and enable this feature when it's
available.
Ensure that we don't overlook a dirty state update that is concurrent
with a call to pmap_enter(). (Previously, all dirty state updates would
have occurred with the containing pmap's lock held, so a page table entry's
dirty state could not have changed while pmap_enter() held that same lock.)
Reviewed by: andrew, markj
MFC after:2 weeks
Differential Revision:https://reviews.freebsd.org/D22907
Modified:
head/sys/arm64/arm64/locore.S
head/sys/arm64/arm64/pmap.c
head/sys/arm64/include/armreg.h
Modified: head/sys/arm64/arm64/locore.S
==
--- head/sys/arm64/arm64/locore.S Mon Dec 30 18:11:06 2019
(r356206)
+++ head/sys/arm64/arm64/locore.S Mon Dec 30 20:30:31 2019
(r356207)
@@ -683,7 +683,8 @@ start_mmu:
/*
* Setup TCR according to the PARange and ASIDBits fields
-* from ID_AA64MMFR0_EL1. More precisely, set TCR_EL1.AS
+* from ID_AA64MMFR0_EL1 and the HAFDBS field from the
+* ID_AA64MMFR1_EL1. More precisely, set TCR_EL1.AS
* to 1 only if the ASIDBits field equals 0b0010.
*/
ldr x2, tcr
@@ -700,6 +701,21 @@ start_mmu:
/* Set TCR.AS with x3 */
bfi x2, x3, #(TCR_ASID_SHIFT), #(TCR_ASID_WIDTH)
+ /*
+* Check if the HW supports access flag and dirty state updates,
+* and set TCR_EL1.HA and TCR_EL1.HD accordingly.
+*/
+ mrs x3, id_aa64mmfr1_el1
+ and x3, x3, #(ID_AA64MMFR1_HAFDBS_MASK)
+ cmp x3, #1
+ b.ne1f
+ orr x2, x2, #(TCR_HA)
+ b 2f
+1:
+ cmp x3, #2
+ b.ne2f
+ orr x2, x2, #(TCR_HA | TCR_HD)
+2:
msr tcr_el1, x2
/*
Modified: head/sys/arm64/arm64/pmap.c
==
--- head/sys/arm64/arm64/pmap.c Mon Dec 30 18:11:06 2019(r356206)
+++ head/sys/arm64/arm64/pmap.c Mon Dec 30 20:30:31 2019(r356207)
@@ -3510,8 +3510,7 @@ validate:
KASSERT(opa == pa, ("pmap_enter: invalid update"));
if ((orig_l3 & ~ATTR_AF) != (new_l3 & ~ATTR_AF)) {
/* same PA, different attributes */
- /* XXXMJ need to reload orig_l3 for hardware DBM. */
- pmap_load_store(l3, new_l3);
+ orig_l3 = pmap_load_store(l3, new_l3);
pmap_invalidate_page(pmap, va);
if ((orig_l3 & ATTR_SW_MANAGED) != 0 &&
pmap_pte_dirty(orig_l3))
Modified: head/sys/arm64/include/armreg.h
==
--- head/sys/arm64/include/armreg.h Mon Dec 30 18:11:06 2019
(r356206)
+++ head/sys/arm64/include/armreg.h Mon Dec 30 20:30:31 2019
(r356207)
@@ -619,6 +619,11 @@
#definePSR_FLAGS 0xf000
/* TCR_EL1 - Translation Control Register */
+#defineTCR_HD_SHIFT40
+#defineTCR_HD (0x1UL << TCR_HD_SHIFT)
+#defineTCR_HA_SHIFT39
+#defineTCR_HA (0x1UL << TCR_HA_SHIFT)
+
#defineTCR_ASID_SHIFT 36
#defineTCR_ASID_WIDTH 1
#defineTCR_ASID_16 (0x1UL << TCR_ASID_SHIFT)
___
svn-src-all@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-all
To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"
svn commit: r356168 - head/sys/i386/i386
Author: alc
Date: Sun Dec 29 05:36:01 2019
New Revision: 356168
URL: https://svnweb.freebsd.org/changeset/base/356168
Log:
Correctly implement PMAP_ENTER_NOREPLACE in pmap_enter_{l2,pde}() on kernel
mappings.
Reduce code duplication by defining a function, pmap_abort_ptp(), for
handling a common error case.
Simplify error handling in pmap_enter_quick_locked().
Reviewed by: kib
Tested by:pho
MFC after:1 week
Differential Revision:https://reviews.freebsd.org/D22890
Modified:
head/sys/i386/i386/pmap.c
Modified: head/sys/i386/i386/pmap.c
==
--- head/sys/i386/i386/pmap.c Sun Dec 29 04:58:27 2019(r356167)
+++ head/sys/i386/i386/pmap.c Sun Dec 29 05:36:01 2019(r356168)
@@ -314,6 +314,7 @@ static pv_entry_t pmap_pvh_remove(struct md_page *pvh,
vm_offset_t va);
static int pmap_pvh_wired_mappings(struct md_page *pvh, int count);
+static voidpmap_abort_ptp(pmap_t pmap, vm_offset_t va, vm_page_t mpte);
static boolean_t pmap_demote_pde(pmap_t pmap, pd_entry_t *pde, vm_offset_t va);
static boolpmap_enter_4mpage(pmap_t pmap, vm_offset_t va, vm_page_t m,
vm_prot_t prot);
@@ -2008,6 +2009,27 @@ pmap_unuse_pt(pmap_t pmap, vm_offset_t va, struct spgl
}
/*
+ * Release a page table page reference after a failed attempt to create a
+ * mapping.
+ */
+static void
+pmap_abort_ptp(pmap_t pmap, vm_offset_t va, vm_page_t mpte)
+{
+ struct spglist free;
+
+ SLIST_INIT();
+ if (pmap_unwire_ptp(pmap, mpte, )) {
+ /*
+* Although "va" was never mapped, paging-structure caches
+* could nonetheless have entries that refer to the freed
+* page table pages. Invalidate those entries.
+*/
+ pmap_invalidate_page_int(pmap, va);
+ vm_page_free_pages_toq(, true);
+ }
+}
+
+/*
* Initialize the pmap for the swapper process.
*/
static void
@@ -3895,6 +3917,24 @@ pmap_enter_4mpage(pmap_t pmap, vm_offset_t va, vm_page
}
/*
+ * Returns true if every page table entry in the page table page that maps
+ * the specified kernel virtual address is zero.
+ */
+static bool
+pmap_every_pte_zero(vm_offset_t va)
+{
+ pt_entry_t *pt_end, *pte;
+
+ KASSERT((va & PDRMASK) == 0, ("va is misaligned"));
+ pte = vtopte(va);
+ for (pt_end = pte + NPTEPG; pte < pt_end; pte++) {
+ if (*pte != 0)
+ return (false);
+ }
+ return (true);
+}
+
+/*
* Tries to create the specified 2 or 4 MB page mapping. Returns KERN_SUCCESS
* if the mapping was created, and either KERN_FAILURE or
* KERN_RESOURCE_SHORTAGE otherwise. Returns KERN_FAILURE if
@@ -3921,7 +3961,9 @@ pmap_enter_pde(pmap_t pmap, vm_offset_t va, pd_entry_t
pde = pmap_pde(pmap, va);
oldpde = *pde;
if ((oldpde & PG_V) != 0) {
- if ((flags & PMAP_ENTER_NOREPLACE) != 0) {
+ if ((flags & PMAP_ENTER_NOREPLACE) != 0 && (pmap !=
+ kernel_pmap || (oldpde & PG_PS) != 0 ||
+ !pmap_every_pte_zero(va))) {
CTR2(KTR_PMAP, "pmap_enter_pde: failure for va %#lx"
" in pmap %p", va, pmap);
return (KERN_FAILURE);
@@ -3940,8 +3982,14 @@ pmap_enter_pde(pmap_t pmap, vm_offset_t va, pd_entry_t
if (pmap_remove_ptes(pmap, va, va + NBPDR, ))
pmap_invalidate_all_int(pmap);
}
- vm_page_free_pages_toq(, true);
- if (pmap == kernel_pmap) {
+ if (pmap != kernel_pmap) {
+ vm_page_free_pages_toq(, true);
+ KASSERT(*pde == 0, ("pmap_enter_pde: non-zero pde %p",
+ pde));
+ } else {
+ KASSERT(SLIST_EMPTY(),
+ ("pmap_enter_pde: freed kernel page table page"));
+
/*
* Both pmap_remove_pde() and pmap_remove_ptes() will
* leave the kernel page table page zero filled.
@@ -3949,9 +3997,7 @@ pmap_enter_pde(pmap_t pmap, vm_offset_t va, pd_entry_t
mt = PHYS_TO_VM_PAGE(*pde & PG_FRAME);
if (pmap_insert_pt_page(pmap, mt, false))
panic("pmap_enter_pde: trie insert failed");
- } else
- KASSERT(*pde == 0, ("pmap_enter_pde: non-zero pde %p",
- pde));
+ }
}
if ((newpde & PG_MANAGED) != 0) {
/*
@@ -3982,8 +4028,8 @@ pmap_enter_pde(pmap_t pmap, vm_offset_t va, pd_entry_t
pde_store(pde, newpde);
pmap_pde_mappings++;
- CTR2(KTR_PMAP, "pmap_enter_pde: success for va %#lx"
-
svn commit: r355991 - head/sys/amd64/amd64
Author: alc
Date: Sat Dec 21 22:32:24 2019
New Revision: 355991
URL: https://svnweb.freebsd.org/changeset/base/355991
Log:
Micro-optimize the control flow in _pmap_unwire_ptp(), and eliminate
unnecessary parentheses.
Reviewed by: kib, markj
MFC after:1 week
Differential Revision:https://reviews.freebsd.org/D22893
Modified:
head/sys/amd64/amd64/pmap.c
Modified: head/sys/amd64/amd64/pmap.c
==
--- head/sys/amd64/amd64/pmap.c Sat Dec 21 21:05:53 2019(r355990)
+++ head/sys/amd64/amd64/pmap.c Sat Dec 21 22:32:24 2019(r355991)
@@ -3542,7 +3542,7 @@ _pmap_unwire_ptp(pmap_t pmap, vm_offset_t va, vm_page_
/*
* unmap the page table page
*/
- if (m->pindex >= (NUPDE + NUPDPE)) {
+ if (m->pindex >= NUPDE + NUPDPE) {
/* PDP page */
pml4_entry_t *pml4;
pml4 = pmap_pml4e(pmap, va);
@@ -3569,8 +3569,7 @@ _pmap_unwire_ptp(pmap_t pmap, vm_offset_t va, vm_page_
pdpg = PHYS_TO_VM_PAGE(*pmap_pdpe(pmap, va) & PG_FRAME);
pmap_unwire_ptp(pmap, va, pdpg, free);
- }
- if (m->pindex >= NUPDE && m->pindex < (NUPDE + NUPDPE)) {
+ } else if (m->pindex < NUPDE + NUPDPE) {
/* We just released a PD, unhold the matching PDP */
vm_page_t pdppg;
___
svn-src-all@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-all
To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"
svn commit: r355946 - in head/sys: amd64/amd64 arm64/arm64
Author: alc
Date: Fri Dec 20 20:46:26 2019
New Revision: 355946
URL: https://svnweb.freebsd.org/changeset/base/355946
Log:
Correct a mistakenly inverted condition in r355833.
Noticed by: kib
X-MFC with: r355833
Modified:
head/sys/amd64/amd64/pmap.c
head/sys/arm64/arm64/pmap.c
Modified: head/sys/amd64/amd64/pmap.c
==
--- head/sys/amd64/amd64/pmap.c Fri Dec 20 20:46:11 2019(r355945)
+++ head/sys/amd64/amd64/pmap.c Fri Dec 20 20:46:26 2019(r355946)
@@ -6336,7 +6336,7 @@ pmap_enter_pde(pmap_t pmap, vm_offset_t va, pd_entry_t
("pmap_enter_pde: pdpg's reference count is too low"));
if ((flags & PMAP_ENTER_NOREPLACE) != 0 && (va <
VM_MAXUSER_ADDRESS || (oldpde & PG_PS) != 0 ||
- pmap_every_pte_zero(oldpde & PG_FRAME))) {
+ !pmap_every_pte_zero(oldpde & PG_FRAME))) {
if (pdpg != NULL)
pdpg->ref_count--;
CTR2(KTR_PMAP, "pmap_enter_pde: failure for va %#lx"
Modified: head/sys/arm64/arm64/pmap.c
==
--- head/sys/arm64/arm64/pmap.c Fri Dec 20 20:46:11 2019(r355945)
+++ head/sys/arm64/arm64/pmap.c Fri Dec 20 20:46:26 2019(r355946)
@@ -3644,7 +3644,7 @@ pmap_enter_l2(pmap_t pmap, vm_offset_t va, pd_entry_t
("pmap_enter_l2: l2pg's ref count is too low"));
if ((flags & PMAP_ENTER_NOREPLACE) != 0 && (va <
VM_MAXUSER_ADDRESS || (old_l2 & ATTR_DESCR_MASK) ==
- L2_BLOCK || pmap_every_pte_zero(old_l2 & ~ATTR_MASK))) {
+ L2_BLOCK || !pmap_every_pte_zero(old_l2 & ~ATTR_MASK))) {
if (l2pg != NULL)
l2pg->ref_count--;
CTR2(KTR_PMAP, "pmap_enter_l2: failure for va %#lx"
___
svn-src-all@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-all
To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"
svn commit: r355883 - in head/sys: amd64/amd64 arm64/arm64
Author: alc
Date: Wed Dec 18 18:21:39 2019
New Revision: 355883
URL: https://svnweb.freebsd.org/changeset/base/355883
Log:
When pmap_enter_{l2,pde}() are called to create a kernel mapping, they are
incrementing (and decrementing) the ref_count on kernel page table pages.
They should not do this. Kernel page table pages are expected to have a
fixed ref_count. Address this problem by refactoring pmap_alloc{_l2,pde}()
and their callers. This also eliminates some duplicated code from the
callers.
Correctly implement PMAP_ENTER_NOREPLACE in pmap_enter_{l2,pde}() on kernel
mappings.
Reduce code duplication by defining a function, pmap_abort_ptp(), for
handling a common error case.
Handle a possible page table page leak in pmap_copy(). Suppose that we are
determining whether to copy a superpage mapping. If we abort because there
is already a mapping in the destination pmap at the current address, then
simply decrementing the page table page's ref_count is correct, because the
page table page must have a ref_count > 1. However, if we abort because we
failed to allocate a PV entry, this might be a just allocated page table
page that has a ref_count = 1, so we should call pmap_abort_ptp().
Simplify error handling in pmap_enter_quick_locked().
Reviewed by: kib, markj (an earlier)
MFC after:2 weeks
Differential Revision:https://reviews.freebsd.org/D22763
Modified:
head/sys/amd64/amd64/pmap.c
head/sys/arm64/arm64/pmap.c
Modified: head/sys/amd64/amd64/pmap.c
==
--- head/sys/amd64/amd64/pmap.c Wed Dec 18 16:01:15 2019(r355882)
+++ head/sys/amd64/amd64/pmap.c Wed Dec 18 18:21:39 2019(r355883)
@@ -1202,6 +1202,7 @@ static void pmap_pvh_free(struct md_page *pvh,
pmap_t
static pv_entry_t pmap_pvh_remove(struct md_page *pvh, pmap_t pmap,
vm_offset_t va);
+static voidpmap_abort_ptp(pmap_t pmap, vm_offset_t va, vm_page_t mpte);
static int pmap_change_props_locked(vm_offset_t va, vm_size_t size,
vm_prot_t prot, int mode, int flags);
static boolean_t pmap_demote_pde(pmap_t pmap, pd_entry_t *pde, vm_offset_t va);
@@ -1256,7 +1257,7 @@ static void pmap_update_pde_invalidate(pmap_t, vm_offs
static vm_page_t _pmap_allocpte(pmap_t pmap, vm_pindex_t ptepindex,
struct rwlock **lockp);
-static vm_page_t pmap_allocpde(pmap_t pmap, vm_offset_t va,
+static pd_entry_t *pmap_alloc_pde(pmap_t pmap, vm_offset_t va, vm_page_t
*pdpgp,
struct rwlock **lockp);
static vm_page_t pmap_allocpte(pmap_t pmap, vm_offset_t va,
struct rwlock **lockp);
@@ -3601,6 +3602,27 @@ pmap_unuse_pt(pmap_t pmap, vm_offset_t va, pd_entry_t
return (pmap_unwire_ptp(pmap, va, mpte, free));
}
+/*
+ * Release a page table page reference after a failed attempt to create a
+ * mapping.
+ */
+static void
+pmap_abort_ptp(pmap_t pmap, vm_offset_t va, vm_page_t mpte)
+{
+ struct spglist free;
+
+ SLIST_INIT();
+ if (pmap_unwire_ptp(pmap, va, mpte, )) {
+ /*
+* Although "va" was never mapped, paging-structure caches
+* could nonetheless have entries that refer to the freed
+* page table pages. Invalidate those entries.
+*/
+ pmap_invalidate_page(pmap, va);
+ vm_page_free_pages_toq(, true);
+ }
+}
+
void
pmap_pinit0(pmap_t pmap)
{
@@ -3906,30 +3928,44 @@ _pmap_allocpte(pmap_t pmap, vm_pindex_t ptepindex, str
return (m);
}
-static vm_page_t
-pmap_allocpde(pmap_t pmap, vm_offset_t va, struct rwlock **lockp)
+static pd_entry_t *
+pmap_alloc_pde(pmap_t pmap, vm_offset_t va, vm_page_t *pdpgp,
+struct rwlock **lockp)
{
- vm_pindex_t pdpindex, ptepindex;
pdp_entry_t *pdpe, PG_V;
+ pd_entry_t *pde;
vm_page_t pdpg;
+ vm_pindex_t pdpindex;
PG_V = pmap_valid_bit(pmap);
retry:
pdpe = pmap_pdpe(pmap, va);
if (pdpe != NULL && (*pdpe & PG_V) != 0) {
- /* Add a reference to the pd page. */
- pdpg = PHYS_TO_VM_PAGE(*pdpe & PG_FRAME);
- pdpg->ref_count++;
- } else {
+ pde = pmap_pdpe_to_pde(pdpe, va);
+ if (va < VM_MAXUSER_ADDRESS) {
+ /* Add a reference to the pd page. */
+ pdpg = PHYS_TO_VM_PAGE(*pdpe & PG_FRAME);
+ pdpg->ref_count++;
+ } else
+ pdpg = NULL;
+ } else if (va < VM_MAXUSER_ADDRESS) {
/* Allocate a pd page. */
- ptepindex = pmap_pde_pindex(va);
- pdpindex = ptepindex >> NPDPEPGSHIFT;
+ pdpindex = pmap_pde_pindex(va) >> NPDPEPGSHIFT;
pdpg = _pmap_allocpte(pmap, NUPDE + pdpindex, lockp);
- if (pdpg == NULL && lockp != NULL)
-
svn commit: r355787 - head/sys/arm64/arm64
Author: alc
Date: Sun Dec 15 22:41:57 2019
New Revision: 355787
URL: https://svnweb.freebsd.org/changeset/base/355787
Log:
Apply a small optimization to pmap_remove_l3_range(). Specifically, hoist a
PHYS_TO_VM_PAGE() operation that always returns the same vm_page_t out of
the loop. (Since arm64 is configured as VM_PHYSSEG_SPARSE, the
implementation of PHYS_TO_VM_PAGE() is more costly than that of
VM_PHYSSEG_DENSE platforms, like amd64.)
MFC after:1 week
Modified:
head/sys/arm64/arm64/pmap.c
Modified: head/sys/arm64/arm64/pmap.c
==
--- head/sys/arm64/arm64/pmap.c Sun Dec 15 21:52:40 2019(r355786)
+++ head/sys/arm64/arm64/pmap.c Sun Dec 15 22:41:57 2019(r355787)
@@ -2616,11 +2616,13 @@ pmap_remove_l3_range(pmap_t pmap, pd_entry_t l2e, vm_o
struct rwlock *new_lock;
pt_entry_t *l3, old_l3;
vm_offset_t va;
- vm_page_t m;
+ vm_page_t l3pg, m;
PMAP_LOCK_ASSERT(pmap, MA_OWNED);
KASSERT(rounddown2(sva, L2_SIZE) + L2_SIZE == roundup2(eva, L2_SIZE),
("pmap_remove_l3_range: range crosses an L3 page table boundary"));
+ l3pg = sva < VM_MAXUSER_ADDRESS ? PHYS_TO_VM_PAGE(l2e & ~ATTR_MASK) :
+ NULL;
va = eva;
for (l3 = pmap_l2_to_l3(, sva); sva != eva; l3++, sva += L3_SIZE) {
if (!pmap_l3_valid(pmap_load(l3))) {
@@ -2671,7 +2673,7 @@ pmap_remove_l3_range(pmap_t pmap, pd_entry_t l2e, vm_o
}
if (va == eva)
va = sva;
- if (pmap_unuse_pt(pmap, sva, l2e, free)) {
+ if (l3pg != NULL && pmap_unwire_l3(pmap, sva, l3pg, free)) {
sva += L3_SIZE;
break;
}
___
svn-src-all@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-all
To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"
svn commit: r355427 - head/sys/arm64/arm64
Author: alc
Date: Thu Dec 5 19:25:49 2019
New Revision: 355427
URL: https://svnweb.freebsd.org/changeset/base/355427
Log:
On a context switch, handle the possibility that the old thread was
preempted after an "ic" or "tlbi" instruction but before it performed a
"dsb" instruction. The "ic" and "tlbi" instructions have unusual
synchronization requirements. If the old thread migrates to a new
processor, its completion of a "dsb" instruction on that new processor does
not guarantee that the "ic" or "tlbi" instructions performed on the old
processor have completed.
This issue is not restricted to the kernel. Since locore.S sets the UCI bit
in SCTLR, user-space programs can perform "ic ivau" instructions (as well as
some forms of the "dc" instruction).
Reviewed by: andrew, kib, markj, mmel
X-MFC with: r355145
Differential Revision:https://reviews.freebsd.org/D22622
Modified:
head/sys/arm64/arm64/pmap.c
Modified: head/sys/arm64/arm64/pmap.c
==
--- head/sys/arm64/arm64/pmap.c Thu Dec 5 18:47:29 2019(r355426)
+++ head/sys/arm64/arm64/pmap.c Thu Dec 5 19:25:49 2019(r355427)
@@ -5850,8 +5850,18 @@ pmap_activate_int(pmap_t pmap)
KASSERT(PCPU_GET(curpmap) != NULL, ("no active pmap"));
KASSERT(pmap != kernel_pmap, ("kernel pmap activation"));
- if (pmap == PCPU_GET(curpmap))
+ if (pmap == PCPU_GET(curpmap)) {
+ /*
+* Handle the possibility that the old thread was preempted
+* after an "ic" or "tlbi" instruction but before it performed
+* a "dsb" instruction. If the old thread migrates to a new
+* processor, its completion of a "dsb" instruction on that
+* new processor does not guarantee that the "ic" or "tlbi"
+* instructions performed on the old processor have completed.
+*/
+ dsb(ish);
return (false);
+ }
/*
* Ensure that the store to curpmap is globally visible before the
___
svn-src-all@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-all
To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"
Re: svn commit: r355145 - head/sys/arm64/arm64
On 11/28/19 7:52 AM, Konstantin Belousov wrote:
On Thu, Nov 28, 2019 at 09:17:15AM +, Andrew Turner wrote:
On 28 Nov 2019, at 08:48, Michal Meloun wrote:
On 27.11.2019 21:33, Alan Cox wrote:
Author: alc
Date: Wed Nov 27 20:33:49 2019
New Revision: 355145
URL: https://svnweb.freebsd.org/changeset/base/355145
Log:
There is no reason why we need to pin the underlying thread to its current
processor in pmap_invalidate_{all,page,range}(). These functions are using
an instruction that broadcasts the TLB invalidation to every processor, so
even if a thread migrates in the middle of one of these functions every
processor will still perform the required TLB invalidations.
I think this is not the right assumption. The problem is not in TLB
operations themselves, but in following 'dsb' and / or 'isb'. 'dsb'
ensures that all TLB operation transmitted by the local CPU is performed
and visible to other observers. But it does nothing with TLBs emitted by
other CPUs.
For example, if a given thread is rescheduled after all TLB operations
but before 'dsb' or 'isb' is performed, then the requested
synchronization does not occur at all.
The tibi instructions need a context synchronisation point. One option is the dsb
& isb instructions, another is an exception entry.
For a thread to be rescheduled it requires the timer interrupt to fire. As an
exception entry is a context synchronisation point and an interrupt will cause
an exception entry there will be such a point after the the tibi instruction.
D5.10.2. TLB maintenance instructions, 'Ordering and completion of
TLB maintenance instructions' states that DSB on the PE that issued
TLBI is required. It does not state that arbitrary even causing
SynchronizeContext() is enough.
Also I was not able to find any explanation of SynchronizeContext().
The entry for "Context Synchronization Events" in the Glossary on page
7460 provides the best explanation that I've found. The first three
listed events are an isb instruction, taking an exception, and returning
from an exception. However, there is nothing here to suggest that
taking or returning from an exception can take the place of a dsb
instruction.
(On a related note, I'll observe that Linux does not perform an isb
instruction during TLB invalidation unless it is changing a leaf in the
kernel page table. For changes to user-space page tables, they appear
to be assuming that the return to user-space will suffice to resync the
user-space instruction stream.)
Curiously, on IA32 exceptions are not specified to issue a serialization
point, although rumors say that on all produced microarchitectures they are.
This issue has similarities to the one that we discussed in
https://reviews.freebsd.org/D22007. For example, on a context switch we
will perform a dsb instruction in pmap_activate_int() unless we are
switching to a thread within the same address space. Moreover, that dsb
instruction will be performed before the lock on the old thread is
released. So, in this case, we are guaranteed that any previously
issued tlbi instructions are completed before the thread can be
restarted on another processor.
However, if we are simply switching between threads within the same
address space, we are not performing a dsb instruction. And, my error
was believing that the memory barriers inherent to the locking
operations on the thread during context switches would be sufficient to
ensure that all of the tlbi instructions on the initial processor would
have completed before the dsb instruction on the eventual processor
finishing the pmap_invalidate_*() completed.
After further reading, I'm afraid that we have a similar issue with
cache management functions, like cpu_icache_sync_range(). Quoting
K11.5.2, "The ARMv8 architecture requires a PE that executes an
instruction cache maintenance instruction to execute a DSB instruction
to ensure completion of the maintenance operation." So, we have a
similar problem if we are preempted during the cpu_icache_sync_range()
calls from pmap_enter(), pmap_enter_quick(), and pmap_sync_icache().
Unless we are switching to a different address space, we are not
guaranteed to perform a dsb instruction on the initial processor.
Moreover, we are configuring the processor in locore.S so that
user-space can directly perform "ic ivau" instructions. (See SCTLR_UCI
in sctlr_set.) So, I'm inclined to say that we should handle this issue
analogously to r354630 on amd64:
Index: arm64/arm64/pmap.c
===
--- arm64/arm64/pmap.c (revision 355145)
+++ arm64/arm64/pmap.c (working copy)
@@ -5853,8 +5853,11 @@ pmap_activate_int(pmap_t pmap)
KASSERT(PCPU_GET(curpmap) != NULL, ("no active pmap"));
KASSERT(pmap != kernel_pmap, ("kernel pmap activation"));
- if (pmap == PCPU_GET(curpmap))
+ if (pmap
svn commit: r355145 - head/sys/arm64/arm64
Author: alc
Date: Wed Nov 27 20:33:49 2019
New Revision: 355145
URL: https://svnweb.freebsd.org/changeset/base/355145
Log:
There is no reason why we need to pin the underlying thread to its current
processor in pmap_invalidate_{all,page,range}(). These functions are using
an instruction that broadcasts the TLB invalidation to every processor, so
even if a thread migrates in the middle of one of these functions every
processor will still perform the required TLB invalidations.
Reviewed by: andrew, markj
MFC after:10 days
Differential Revision:https://reviews.freebsd.org/D22502
Modified:
head/sys/arm64/arm64/pmap.c
Modified: head/sys/arm64/arm64/pmap.c
==
--- head/sys/arm64/arm64/pmap.c Wed Nov 27 20:32:53 2019(r355144)
+++ head/sys/arm64/arm64/pmap.c Wed Nov 27 20:33:49 2019(r355145)
@@ -1043,7 +1043,6 @@ pmap_invalidate_page(pmap_t pmap, vm_offset_t va)
{
uint64_t r;
- sched_pin();
dsb(ishst);
if (pmap == kernel_pmap) {
r = atop(va);
@@ -1054,11 +1053,10 @@ pmap_invalidate_page(pmap_t pmap, vm_offset_t va)
}
dsb(ish);
isb();
- sched_unpin();
}
static __inline void
-pmap_invalidate_range_nopin(pmap_t pmap, vm_offset_t sva, vm_offset_t eva)
+pmap_invalidate_range(pmap_t pmap, vm_offset_t sva, vm_offset_t eva)
{
uint64_t end, r, start;
@@ -1080,20 +1078,10 @@ pmap_invalidate_range_nopin(pmap_t pmap, vm_offset_t s
}
static __inline void
-pmap_invalidate_range(pmap_t pmap, vm_offset_t sva, vm_offset_t eva)
-{
-
- sched_pin();
- pmap_invalidate_range_nopin(pmap, sva, eva);
- sched_unpin();
-}
-
-static __inline void
pmap_invalidate_all(pmap_t pmap)
{
uint64_t r;
- sched_pin();
dsb(ishst);
if (pmap == kernel_pmap) {
__asm __volatile("tlbi vmalle1is");
@@ -1103,7 +1091,6 @@ pmap_invalidate_all(pmap_t pmap)
}
dsb(ish);
isb();
- sched_unpin();
}
/*
@@ -3114,7 +3101,7 @@ pmap_update_entry(pmap_t pmap, pd_entry_t *pte, pd_ent
* lookup the physical address.
*/
pmap_clear_bits(pte, ATTR_DESCR_VALID);
- pmap_invalidate_range_nopin(pmap, va, va + size);
+ pmap_invalidate_range(pmap, va, va + size);
/* Create the new mapping */
pmap_store(pte, newpte);
___
svn-src-all@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-all
To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"
svn commit: r355136 - in stable/12/sys/arm64: arm64 include
Author: alc
Date: Wed Nov 27 19:34:33 2019
New Revision: 355136
URL: https://svnweb.freebsd.org/changeset/base/355136
Log:
MFC r350579,r350741,r352584
Enable superpage promotion within the kernel pmap.
Ordinarily, during a superpage promotion or demotion within a pmap, the
pmap's lock ensures that other operations on the pmap don't observe the
old mapping being broken before the new mapping is established. However,
pmap_kextract() doesn't acquire the kernel pmap's lock, so it may observe
the broken mapping. And, if it does, it returns an incorrect result.
This revision implements a lock-free solution to this problem in
pmap_update_entry() and pmap_kextract() because pmap_kextract() can't
acquire the kernel pmap's lock.
In case a translation fault on the kernel address space occurs from
within a critical section, we must perform a lock-free check on the
faulting address.
Modified:
stable/12/sys/arm64/arm64/pmap.c
stable/12/sys/arm64/include/pte.h
Directory Properties:
stable/12/ (props changed)
Modified: stable/12/sys/arm64/arm64/pmap.c
==
--- stable/12/sys/arm64/arm64/pmap.cWed Nov 27 19:32:29 2019
(r355135)
+++ stable/12/sys/arm64/arm64/pmap.cWed Nov 27 19:34:33 2019
(r355136)
@@ -1128,40 +1128,35 @@ vm_paddr_t
pmap_kextract(vm_offset_t va)
{
pt_entry_t *pte, tpte;
- vm_paddr_t pa;
- int lvl;
- if (va >= DMAP_MIN_ADDRESS && va < DMAP_MAX_ADDRESS) {
- pa = DMAP_TO_PHYS(va);
- } else {
- pa = 0;
- pte = pmap_pte(kernel_pmap, va, );
- if (pte != NULL) {
- tpte = pmap_load(pte);
- pa = tpte & ~ATTR_MASK;
- switch(lvl) {
- case 1:
- KASSERT((tpte & ATTR_DESCR_MASK) == L1_BLOCK,
- ("pmap_kextract: Invalid L1 pte found: %lx",
- tpte & ATTR_DESCR_MASK));
- pa |= (va & L1_OFFSET);
- break;
- case 2:
- KASSERT((tpte & ATTR_DESCR_MASK) == L2_BLOCK,
- ("pmap_kextract: Invalid L2 pte found: %lx",
- tpte & ATTR_DESCR_MASK));
- pa |= (va & L2_OFFSET);
- break;
- case 3:
- KASSERT((tpte & ATTR_DESCR_MASK) == L3_PAGE,
- ("pmap_kextract: Invalid L3 pte found: %lx",
- tpte & ATTR_DESCR_MASK));
- pa |= (va & L3_OFFSET);
- break;
- }
- }
- }
- return (pa);
+ if (va >= DMAP_MIN_ADDRESS && va < DMAP_MAX_ADDRESS)
+ return (DMAP_TO_PHYS(va));
+ pte = pmap_l1(kernel_pmap, va);
+ if (pte == NULL)
+ return (0);
+
+ /*
+* A concurrent pmap_update_entry() will clear the entry's valid bit
+* but leave the rest of the entry unchanged. Therefore, we treat a
+* non-zero entry as being valid, and we ignore the valid bit when
+* determining whether the entry maps a block, page, or table.
+*/
+ tpte = pmap_load(pte);
+ if (tpte == 0)
+ return (0);
+ if ((tpte & ATTR_DESCR_TYPE_MASK) == ATTR_DESCR_TYPE_BLOCK)
+ return ((tpte & ~ATTR_MASK) | (va & L1_OFFSET));
+ pte = pmap_l1_to_l2(, va);
+ tpte = pmap_load(pte);
+ if (tpte == 0)
+ return (0);
+ if ((tpte & ATTR_DESCR_TYPE_MASK) == ATTR_DESCR_TYPE_BLOCK)
+ return ((tpte & ~ATTR_MASK) | (va & L2_OFFSET));
+ pte = pmap_l2_to_l3(, va);
+ tpte = pmap_load(pte);
+ if (tpte == 0)
+ return (0);
+ return ((tpte & ~ATTR_MASK) | (va & L3_OFFSET));
}
/***
@@ -3020,8 +3015,12 @@ pmap_update_entry(pmap_t pmap, pd_entry_t *pte, pd_ent
*/
intr = intr_disable();
- /* Clear the old mapping */
- pmap_clear(pte);
+ /*
+* Clear the old mapping's valid bit, but leave the rest of the entry
+* unchanged, so that a lockless, concurrent pmap_kextract() can still
+* lookup the physical address.
+*/
+ pmap_clear_bits(pte, ATTR_DESCR_VALID);
pmap_invalidate_range_nopin(pmap, va, va + size);
/* Create the new mapping */
@@ -3423,8 +3422,7 @@ validate:
}
#if VM_NRESERVLEVEL > 0
- if (pmap != pmap_kernel() &&
- (mpte == NULL || mpte->wire_count == NL3PG) &&
+ if ((mpte == NULL || mpte->wire_count == NL3PG) &&
svn commit: r354898 - head/sys/arm64/arm64
Author: alc Date: Wed Nov 20 16:32:13 2019 New Revision: 354898 URL: https://svnweb.freebsd.org/changeset/base/354898 Log: Until every possible root cause for an "invalid ASID" assertion failure is resolved, assign every pmap a valid ASID when it is first initialized. Modified: head/sys/arm64/arm64/pmap.c Modified: head/sys/arm64/arm64/pmap.c == --- head/sys/arm64/arm64/pmap.c Wed Nov 20 16:30:37 2019(r354897) +++ head/sys/arm64/arm64/pmap.c Wed Nov 20 16:32:13 2019(r354898) @@ -1556,6 +1556,8 @@ pmap_pinit(pmap_t pmap) pmap->pm_root.rt_root = 0; bzero(>pm_stats, sizeof(pmap->pm_stats)); pmap->pm_cookie = COOKIE_FROM(-1, INT_MAX); + /* XXX Temporarily disable deferred ASID allocation. */ + pmap_alloc_asid(pmap); return (1); } ___ svn-src-all@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"
svn commit: r354860 - head/sys/arm64/arm64
Author: alc
Date: Tue Nov 19 19:05:05 2019
New Revision: 354860
URL: https://svnweb.freebsd.org/changeset/base/354860
Log:
Achieve two goals at once: (1) Avoid an unnecessary broadcast TLB
invalidation in reclaim_pv_chunk(). (2) Prevent an "invalid ASID" assertion
failure in reclaim_pv_chunk(). The detailed explanation for this change is
provided by r354792.
X-MFC with: r354792
Modified:
head/sys/arm64/arm64/pmap.c
Modified: head/sys/arm64/arm64/pmap.c
==
--- head/sys/arm64/arm64/pmap.c Tue Nov 19 16:40:46 2019(r354859)
+++ head/sys/arm64/arm64/pmap.c Tue Nov 19 19:05:05 2019(r354860)
@@ -2062,12 +2062,13 @@ reclaim_pv_chunk(pmap_t locked_pmap, struct rwlock **l
if ((tpte & ATTR_SW_WIRED) != 0)
continue;
tpte = pmap_load_clear(pte);
- pmap_invalidate_page(pmap, va);
m = PHYS_TO_VM_PAGE(tpte & ~ATTR_MASK);
if (pmap_pte_dirty(tpte))
vm_page_dirty(m);
- if ((tpte & ATTR_AF) != 0)
+ if ((tpte & ATTR_AF) != 0) {
+ pmap_invalidate_page(pmap, va);
vm_page_aflag_set(m, PGA_REFERENCED);
+ }
CHANGE_PV_LIST_LOCK_TO_VM_PAGE(lockp, m);
TAILQ_REMOVE(>md.pv_list, pv, pv_next);
m->md.pv_gen++;
___
svn-src-all@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-all
To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"
svn commit: r354799 - stable/12/sys/arm64/arm64
Author: alc
Date: Sun Nov 17 22:44:38 2019
New Revision: 354799
URL: https://svnweb.freebsd.org/changeset/base/354799
Log:
MFC r352847,352930,354585
Eliminate redundant calls to critical_enter() and critical_exit() from
pmap_update_entry(). It suffices that interrupts are blocked.
In short, pmap_enter_quick_locked("user space", ..., VM_PROT_READ) doesn't
work. More precisely, it doesn't set ATTR_AP(ATTR_AP_USER) in the page
table entry, so any attempt to read from the mapped page by user space
generates a page fault. This problem has gone unnoticed because the page
fault handler, vm_fault(), will ultimately call pmap_enter(), which
replaces the non-working page table entry with one that has
ATTR_AP(ATTR_AP_USER) set. This change reduces the number of page faults
during a "buildworld" by about 19.4%.
Eliminate a redundant pmap_load() from pmap_remove_pages().
There is no reason why the pmap_invalidate_all() in pmap_remove_pages()
must be performed before the final PV list lock release. Move it past
the lock release.
Eliminate a stale comment from pmap_page_test_mappings(). We implemented
a modified bit in r350004.
Modified:
stable/12/sys/arm64/arm64/pmap.c
Directory Properties:
stable/12/ (props changed)
Modified: stable/12/sys/arm64/arm64/pmap.c
==
--- stable/12/sys/arm64/arm64/pmap.cSun Nov 17 20:56:25 2019
(r354798)
+++ stable/12/sys/arm64/arm64/pmap.cSun Nov 17 22:44:38 2019
(r354799)
@@ -3019,7 +3019,6 @@ pmap_update_entry(pmap_t pmap, pd_entry_t *pte, pd_ent
* as they may make use of an address we are about to invalidate.
*/
intr = intr_disable();
- critical_enter();
/* Clear the old mapping */
pmap_clear(pte);
@@ -3029,7 +3028,6 @@ pmap_update_entry(pmap_t pmap, pd_entry_t *pte, pd_ent
pmap_store(pte, newpte);
dsb(ishst);
- critical_exit();
intr_restore(intr);
}
@@ -3763,8 +3761,8 @@ pmap_enter_quick_locked(pmap_t pmap, vm_offset_t va, v
ATTR_AP(ATTR_AP_RO) | L3_PAGE;
if ((prot & VM_PROT_EXECUTE) == 0 || m->md.pv_memattr == DEVICE_MEMORY)
l3_val |= ATTR_XN;
- else if (va < VM_MAXUSER_ADDRESS)
- l3_val |= ATTR_PXN;
+ if (va < VM_MAXUSER_ADDRESS)
+ l3_val |= ATTR_AP(ATTR_AP_USER) | ATTR_PXN;
/*
* Now validate mapping with RO protection
@@ -4330,7 +4328,6 @@ pmap_remove_pages(pmap_t pmap)
L2_BLOCK,
("Attempting to remove an invalid "
"block: %lx", tpte));
- tpte = pmap_load(pte);
break;
case 2:
pte = pmap_l2_to_l3(pde, pv->pv_va);
@@ -4448,17 +4445,15 @@ pmap_remove_pages(pmap_t pmap)
free_pv_chunk(pc);
}
}
- pmap_invalidate_all(pmap);
if (lock != NULL)
rw_wunlock(lock);
+ pmap_invalidate_all(pmap);
PMAP_UNLOCK(pmap);
vm_page_free_pages_toq(, true);
}
/*
- * This is used to check if a page has been accessed or modified. As we
- * don't have a bit to see if it has been modified we have to assume it
- * has been if the page is read/write.
+ * This is used to check if a page has been accessed or modified.
*/
static boolean_t
pmap_page_test_mappings(vm_page_t m, boolean_t accessed, boolean_t modified)
___
svn-src-all@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-all
To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"
svn commit: r354792 - head/sys/arm64/arm64
Author: alc
Date: Sun Nov 17 17:38:53 2019
New Revision: 354792
URL: https://svnweb.freebsd.org/changeset/base/354792
Log:
Achieve two goals at once: (1) Avoid an unnecessary broadcast TLB
invalidation in pmap_remove_all(). (2) Prevent an "invalid ASID" assertion
failure in pmap_remove_all().
The architecture definition specifies that the TLB will not cache mappings
that don't have the "AF" bit set, so pmap_remove_all() needn't issue a TLB
invalidation for mappings that don't have the "AF" bit set.
We allocate ASIDs lazily. Specifically, we don't allocate an ASID for a
pmap until we are activating it. Now, consider what happens on a fork().
Before we activate the child's pmap, we use pmap_copy() to copy mappings
from the parent's pmap to the child's. These new mappings have their "AF"
bits cleared. Suppose that the page daemon decides to reclaim a page that
underlies one of these new mappings. Previously, the pmap_invalidate_page()
performed by pmap_remove_all() on a mapping in the child's pmap would fail
an assertion because that pmap hasn't yet been assigned an ASID. However,
we don't need to issue a TLB invalidation for such mappings because they
can't possibly be cached in the TLB.
Reported by: bob prohaska
Reviewed by: markj
MFC after:1 week
X-MFC-before: r354286
Differential Revision:https://reviews.freebsd.org/D22388
Modified:
head/sys/arm64/arm64/pmap.c
Modified: head/sys/arm64/arm64/pmap.c
==
--- head/sys/arm64/arm64/pmap.c Sun Nov 17 14:54:43 2019(r354791)
+++ head/sys/arm64/arm64/pmap.c Sun Nov 17 17:38:53 2019(r354792)
@@ -2879,11 +2879,12 @@ retry:
pte = pmap_l2_to_l3(pde, pv->pv_va);
tpte = pmap_load_clear(pte);
- pmap_invalidate_page(pmap, pv->pv_va);
if (tpte & ATTR_SW_WIRED)
pmap->pm_stats.wired_count--;
- if ((tpte & ATTR_AF) != 0)
+ if ((tpte & ATTR_AF) != 0) {
+ pmap_invalidate_page(pmap, pv->pv_va);
vm_page_aflag_set(m, PGA_REFERENCED);
+ }
/*
* Update the vm_page_t clean and reference bits.
___
svn-src-all@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-all
To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"
svn commit: r354585 - head/sys/arm64/arm64
Author: alc
Date: Sun Nov 10 05:22:01 2019
New Revision: 354585
URL: https://svnweb.freebsd.org/changeset/base/354585
Log:
Eliminate a redundant pmap_load() from pmap_remove_pages().
There is no reason why the pmap_invalidate_all() in pmap_remove_pages()
must be performed before the final PV list lock release. Move it past
the lock release.
Eliminate a stale comment from pmap_page_test_mappings(). We implemented
a modified bit in r350004.
MFC after:1 week
Modified:
head/sys/arm64/arm64/pmap.c
Modified: head/sys/arm64/arm64/pmap.c
==
--- head/sys/arm64/arm64/pmap.c Sun Nov 10 04:24:36 2019(r354584)
+++ head/sys/arm64/arm64/pmap.c Sun Nov 10 05:22:01 2019(r354585)
@@ -4434,7 +4434,6 @@ pmap_remove_pages(pmap_t pmap)
L2_BLOCK,
("Attempting to remove an invalid "
"block: %lx", tpte));
- tpte = pmap_load(pte);
break;
case 2:
pte = pmap_l2_to_l3(pde, pv->pv_va);
@@ -4552,17 +4551,15 @@ pmap_remove_pages(pmap_t pmap)
free_pv_chunk(pc);
}
}
- pmap_invalidate_all(pmap);
if (lock != NULL)
rw_wunlock(lock);
+ pmap_invalidate_all(pmap);
PMAP_UNLOCK(pmap);
vm_page_free_pages_toq(, true);
}
/*
- * This is used to check if a page has been accessed or modified. As we
- * don't have a bit to see if it has been modified we have to assume it
- * has been if the page is read/write.
+ * This is used to check if a page has been accessed or modified.
*/
static boolean_t
pmap_page_test_mappings(vm_page_t m, boolean_t accessed, boolean_t modified)
___
svn-src-all@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-all
To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"
svn commit: r354286 - in head/sys/arm64: arm64 include
Author: alc
Date: Sun Nov 3 17:45:30 2019
New Revision: 354286
URL: https://svnweb.freebsd.org/changeset/base/354286
Log:
Utilize ASIDs to reduce both the direct and indirect costs of context
switching. The indirect costs being unnecessary TLB misses that are
incurred when ASIDs are not used. In fact, currently, when we perform a
context switch on one processor, we issue a broadcast TLB invalidation that
flushes the TLB contents on every processor.
Mark all user-space ("ttbr0") page table entries with the non-global flag so
that they are cached in the TLB under their ASID.
Correct an error in pmap_pinit0(). The pointer to the root of the page
table was being initialized to the root of the kernel-space page table
rather than a user-space page table. However, the root of the page table
that was being cached in process 0's md_l0addr field correctly pointed to a
user-space page table. As long as ASIDs weren't being used, this was
harmless, except that it led to some unnecessary page table switches in
pmap_switch(). Specifically, other kernel processes besides process 0 would
have their md_l0addr field set to the root of the kernel-space page table,
and so pmap_switch() would actually change page tables when switching
between process 0 and other kernel processes.
Implement a workaround for Cavium erratum 27456 affecting ThunderX machines.
(I would like to thank andrew@ for providing the code to detect the affected
machines.)
Address integer overflow in the definition of TCR_ASID_16.
Setup TCR according to the PARange and ASIDBits fields from
ID_AA64MMFR0_EL1. Previously, TCR_ASID_16 was unconditionally set.
Modify build_l1_block_pagetable so that lower attributes, such as ATTR_nG,
can be specified as a parameter.
Eliminate some unused code.
Earlier versions were tested to varying degrees by: andrew, emaste, markj
MFC after:3 weeks
Differential Revision:https://reviews.freebsd.org/D21922
Modified:
head/sys/arm64/arm64/cpu_errata.c
head/sys/arm64/arm64/cpufunc_asm.S
head/sys/arm64/arm64/efirt_machdep.c
head/sys/arm64/arm64/genassym.c
head/sys/arm64/arm64/locore.S
head/sys/arm64/arm64/machdep.c
head/sys/arm64/arm64/mp_machdep.c
head/sys/arm64/arm64/pmap.c
head/sys/arm64/arm64/vm_machdep.c
head/sys/arm64/include/armreg.h
head/sys/arm64/include/cpufunc.h
head/sys/arm64/include/pcpu.h
head/sys/arm64/include/pmap.h
head/sys/arm64/include/proc.h
Modified: head/sys/arm64/arm64/cpu_errata.c
==
--- head/sys/arm64/arm64/cpu_errata.c Sun Nov 3 15:42:08 2019
(r354285)
+++ head/sys/arm64/arm64/cpu_errata.c Sun Nov 3 17:45:30 2019
(r354286)
@@ -59,6 +59,7 @@ static enum {
static cpu_quirk_install install_psci_bp_hardening;
static cpu_quirk_install install_ssbd_workaround;
+static cpu_quirk_install install_thunderx_bcast_tlbi_workaround;
static struct cpu_quirks cpu_quirks[] = {
{
@@ -92,6 +93,18 @@ static struct cpu_quirks cpu_quirks[] = {
.midr_value = 0,
.quirk_install = install_ssbd_workaround,
},
+ {
+ .midr_mask = CPU_IMPL_MASK | CPU_PART_MASK,
+ .midr_value =
+ CPU_ID_RAW(CPU_IMPL_CAVIUM, CPU_PART_THUNDERX, 0, 0),
+ .quirk_install = install_thunderx_bcast_tlbi_workaround,
+ },
+ {
+ .midr_mask = CPU_IMPL_MASK | CPU_PART_MASK,
+ .midr_value =
+ CPU_ID_RAW(CPU_IMPL_CAVIUM, CPU_PART_THUNDERX_81XX, 0, 0),
+ .quirk_install = install_thunderx_bcast_tlbi_workaround,
+ },
};
static void
@@ -135,6 +148,30 @@ install_ssbd_workaround(void)
default:
PCPU_SET(ssbd, smccc_arch_workaround_2);
break;
+ }
+}
+
+/*
+ * Workaround Cavium erratum 27456.
+ *
+ * Invalidate the local icache when changing address spaces.
+ */
+static void
+install_thunderx_bcast_tlbi_workaround(void)
+{
+ u_int midr;
+
+ midr = get_midr();
+ if (CPU_PART(midr) == CPU_PART_THUNDERX_81XX)
+ PCPU_SET(bcast_tlbi_workaround, 1);
+ else if (CPU_PART(midr) == CPU_PART_THUNDERX) {
+ if (CPU_VAR(midr) == 0) {
+ /* ThunderX 1.x */
+ PCPU_SET(bcast_tlbi_workaround, 1);
+ } else if (CPU_VAR(midr) == 1 && CPU_REV(midr) <= 1) {
+ /* ThunderX 2.0 - 2.1 */
+ PCPU_SET(bcast_tlbi_workaround, 1);
+ }
}
}
Modified: head/sys/arm64/arm64/cpufunc_asm.S
==
--- head/sys/arm64/arm64/cpufunc_asm.S Sun Nov 3 15:42:08 2019
(r354285)
+++ head/sys/arm64/arm64/cpufunc_asm.S Sun Nov 3 17:45:30 2019
(r354286)
@@ -93,14 +93,6 @@ END(arm64_nullop)
svn commit: r353163 - in head/sys: arm64/include riscv/include
Author: alc Date: Mon Oct 7 04:22:03 2019 New Revision: 353163 URL: https://svnweb.freebsd.org/changeset/base/353163 Log: Eliminate an unused declaration. The variable in question is only defined and used on sparc64. MFC after:1 week Modified: head/sys/arm64/include/vmparam.h head/sys/riscv/include/vmparam.h Modified: head/sys/arm64/include/vmparam.h == --- head/sys/arm64/include/vmparam.hMon Oct 7 03:37:28 2019 (r353162) +++ head/sys/arm64/include/vmparam.hMon Oct 7 04:22:03 2019 (r353163) @@ -234,7 +234,6 @@ extern vm_paddr_t dmap_phys_base; extern vm_paddr_t dmap_phys_max; extern vm_offset_t dmap_max_addr; -extern u_int tsb_kernel_ldd_phys; extern vm_offset_t vm_max_kernel_address; extern vm_offset_t init_pt_va; Modified: head/sys/riscv/include/vmparam.h == --- head/sys/riscv/include/vmparam.hMon Oct 7 03:37:28 2019 (r353162) +++ head/sys/riscv/include/vmparam.hMon Oct 7 04:22:03 2019 (r353163) @@ -228,7 +228,6 @@ extern vm_paddr_t dmap_phys_base; extern vm_paddr_t dmap_phys_max; extern vm_offset_t dmap_max_addr; -extern u_int tsb_kernel_ldd_phys; extern vm_offset_t vm_max_kernel_address; extern vm_offset_t init_pt_va; #endif ___ svn-src-all@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"
svn commit: r353162 - head/sys/arm64/arm64
Author: alc
Date: Mon Oct 7 03:37:28 2019
New Revision: 353162
URL: https://svnweb.freebsd.org/changeset/base/353162
Log:
Eliminate a redundant bzero(). The l0 page table page was already zeroed
by efi_1t1_page().
MFC after:1 week
Modified:
head/sys/arm64/arm64/efirt_machdep.c
Modified: head/sys/arm64/arm64/efirt_machdep.c
==
--- head/sys/arm64/arm64/efirt_machdep.cMon Oct 7 03:28:11 2019
(r353161)
+++ head/sys/arm64/arm64/efirt_machdep.cMon Oct 7 03:37:28 2019
(r353162)
@@ -176,7 +176,6 @@ efi_create_1t1_map(struct efi_md *map, int ndesc, int
efi_l0_page = efi_1t1_page();
VM_OBJECT_WUNLOCK(obj_1t1_pt);
efi_l0 = (pd_entry_t *)PHYS_TO_DMAP(VM_PAGE_TO_PHYS(efi_l0_page));
- bzero(efi_l0, L0_ENTRIES * sizeof(*efi_l0));
for (i = 0, p = map; i < ndesc; i++, p = efi_next_descriptor(p,
descsz)) {
___
svn-src-all@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-all
To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"
svn commit: r353096 - head/sys/arm64/include
Author: alc Date: Fri Oct 4 03:55:53 2019 New Revision: 353096 URL: https://svnweb.freebsd.org/changeset/base/353096 Log: The implementation of arm64_tlb_flushID_SE() was removed from cpufunc_asm.S in r313347. Eliminate its declaration from this file. MFC after:1 week Modified: head/sys/arm64/include/cpufunc.h Modified: head/sys/arm64/include/cpufunc.h == --- head/sys/arm64/include/cpufunc.hFri Oct 4 02:34:20 2019 (r353095) +++ head/sys/arm64/include/cpufunc.hFri Oct 4 03:55:53 2019 (r353096) @@ -200,7 +200,6 @@ extern int64_t dczva_line_size; void arm64_nullop(void); void arm64_setttb(vm_offset_t); void arm64_tlb_flushID(void); -void arm64_tlb_flushID_SE(vm_offset_t); void arm64_icache_sync_range(vm_offset_t, vm_size_t); int arm64_icache_sync_range_checked(vm_offset_t, vm_size_t); void arm64_idcache_wbinv_range(vm_offset_t, vm_size_t); ___ svn-src-all@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"
svn commit: r352930 - head/sys/arm64/arm64
Author: alc
Date: Tue Oct 1 15:33:47 2019
New Revision: 352930
URL: https://svnweb.freebsd.org/changeset/base/352930
Log:
In short, pmap_enter_quick_locked("user space", ..., VM_PROT_READ) doesn't
work. More precisely, it doesn't set ATTR_AP(ATTR_AP_USER) in the page
table entry, so any attempt to read from the mapped page by user space
generates a page fault. This problem has gone unnoticed because the page
fault handler, vm_fault(), will ultimately call pmap_enter(), which
replaces the non-working page table entry with one that has
ATTR_AP(ATTR_AP_USER) set.
This change reduces the number of page faults during a "buildworld" by
about 19.4%.
Reviewed by: andrew, markj
MFC after:1 week
Differential Revision:https://reviews.freebsd.org/D21841
Modified:
head/sys/arm64/arm64/pmap.c
Modified: head/sys/arm64/arm64/pmap.c
==
--- head/sys/arm64/arm64/pmap.c Tue Oct 1 15:10:35 2019(r352929)
+++ head/sys/arm64/arm64/pmap.c Tue Oct 1 15:33:47 2019(r352930)
@@ -3752,8 +3752,8 @@ pmap_enter_quick_locked(pmap_t pmap, vm_offset_t va, v
ATTR_AP(ATTR_AP_RO) | L3_PAGE;
if ((prot & VM_PROT_EXECUTE) == 0 || m->md.pv_memattr == DEVICE_MEMORY)
l3_val |= ATTR_XN;
- else if (va < VM_MAXUSER_ADDRESS)
- l3_val |= ATTR_PXN;
+ if (va < VM_MAXUSER_ADDRESS)
+ l3_val |= ATTR_AP(ATTR_AP_USER) | ATTR_PXN;
/*
* Now validate mapping with RO protection
___
svn-src-all@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-all
To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"
svn commit: r352847 - head/sys/arm64/arm64
Author: alc Date: Sat Sep 28 17:16:03 2019 New Revision: 352847 URL: https://svnweb.freebsd.org/changeset/base/352847 Log: Eliminate redundant calls to critical_enter() and critical_exit() from pmap_update_entry(). It suffices that interrupts are blocked. Reviewed by: andrew, markj MFC after:1 week Differential Revision:https://reviews.freebsd.org/D21753 Modified: head/sys/arm64/arm64/pmap.c Modified: head/sys/arm64/arm64/pmap.c == --- head/sys/arm64/arm64/pmap.c Sat Sep 28 17:15:48 2019(r352846) +++ head/sys/arm64/arm64/pmap.c Sat Sep 28 17:16:03 2019(r352847) @@ -3007,7 +3007,6 @@ pmap_update_entry(pmap_t pmap, pd_entry_t *pte, pd_ent * as they may make use of an address we are about to invalidate. */ intr = intr_disable(); - critical_enter(); /* * Clear the old mapping's valid bit, but leave the rest of the entry @@ -3021,7 +3020,6 @@ pmap_update_entry(pmap_t pmap, pd_entry_t *pte, pd_ent pmap_store(pte, newpte); dsb(ishst); - critical_exit(); intr_restore(intr); } ___ svn-src-all@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"
svn commit: r352584 - head/sys/arm64/arm64
Author: alc
Date: Sat Sep 21 19:51:57 2019
New Revision: 352584
URL: https://svnweb.freebsd.org/changeset/base/352584
Log:
In case a translation fault on the kernel address space occurs from
within a critical section, we must perform a lock-free check on the
faulting address.
Reported by: andrew
Reviewed by: andrew, markj
X-MFC with: r350579
Differential Revision:https://reviews.freebsd.org/D21685
Modified:
head/sys/arm64/arm64/pmap.c
Modified: head/sys/arm64/arm64/pmap.c
==
--- head/sys/arm64/arm64/pmap.c Sat Sep 21 19:05:38 2019(r352583)
+++ head/sys/arm64/arm64/pmap.c Sat Sep 21 19:51:57 2019(r352584)
@@ -5810,23 +5810,33 @@ pmap_fault(pmap_t pmap, uint64_t esr, uint64_t far)
case ISS_DATA_DFSC_TF_L1:
case ISS_DATA_DFSC_TF_L2:
case ISS_DATA_DFSC_TF_L3:
- PMAP_LOCK(pmap);
- /* Ask the MMU to check the address */
- intr = intr_disable();
- if (pmap == kernel_pmap)
- par = arm64_address_translate_s1e1r(far);
- else
- par = arm64_address_translate_s1e0r(far);
- intr_restore(intr);
- PMAP_UNLOCK(pmap);
-
/*
-* If the translation was successful the address was invalid
-* due to a break-before-make sequence. We can unlock and
-* return success to the trap handler.
+* Retry the translation. A break-before-make sequence can
+* produce a transient fault.
*/
- if (PAR_SUCCESS(par))
- rv = KERN_SUCCESS;
+ if (pmap == kernel_pmap) {
+ /*
+* The translation fault may have occurred within a
+* critical section. Therefore, we must check the
+* address without acquiring the kernel pmap's lock.
+*/
+ if (pmap_kextract(far) != 0)
+ rv = KERN_SUCCESS;
+ } else {
+ PMAP_LOCK(pmap);
+ /* Ask the MMU to check the address. */
+ intr = intr_disable();
+ par = arm64_address_translate_s1e0r(far);
+ intr_restore(intr);
+ PMAP_UNLOCK(pmap);
+
+ /*
+* If the translation was successful, then we can
+* return success to the trap handler.
+*/
+ if (PAR_SUCCESS(par))
+ rv = KERN_SUCCESS;
+ }
break;
}
___
svn-src-all@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-all
To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"
svn commit: r352541 - stable/12/sys/amd64/amd64
Author: alc
Date: Thu Sep 19 20:45:23 2019
New Revision: 352541
URL: https://svnweb.freebsd.org/changeset/base/352541
Log:
MF r349585
Tidy up pmap_copy(). Notably, deindent the innermost loop by making a
simple change to the control flow. Replace an unnecessary test by a
KASSERT. Add a comment explaining an obscure test.
Modified:
stable/12/sys/amd64/amd64/pmap.c
Directory Properties:
stable/12/ (props changed)
Modified: stable/12/sys/amd64/amd64/pmap.c
==
--- stable/12/sys/amd64/amd64/pmap.cThu Sep 19 20:16:51 2019
(r352540)
+++ stable/12/sys/amd64/amd64/pmap.cThu Sep 19 20:45:23 2019
(r352541)
@@ -6355,18 +6355,18 @@ pmap_unwire(pmap_t pmap, vm_offset_t sva, vm_offset_t
*
* This routine is only advisory and need not do anything.
*/
-
void
pmap_copy(pmap_t dst_pmap, pmap_t src_pmap, vm_offset_t dst_addr, vm_size_t
len,
vm_offset_t src_addr)
{
struct rwlock *lock;
struct spglist free;
- vm_offset_t addr;
- vm_offset_t end_addr = src_addr + len;
- vm_offset_t va_next;
+ pml4_entry_t *pml4e;
+ pdp_entry_t *pdpe;
+ pd_entry_t *pde, srcptepaddr;
+ pt_entry_t *dst_pte, PG_A, PG_M, PG_V, ptetemp, *src_pte;
+ vm_offset_t addr, end_addr, va_next;
vm_page_t dst_pdpg, dstmpte, srcmpte;
- pt_entry_t PG_A, PG_M, PG_V;
if (dst_addr != src_addr)
return;
@@ -6385,6 +6385,7 @@ pmap_copy(pmap_t dst_pmap, pmap_t src_pmap, vm_offset_
if (pmap_emulate_ad_bits(dst_pmap))
return;
+ end_addr = src_addr + len;
lock = NULL;
if (dst_pmap < src_pmap) {
PMAP_LOCK(dst_pmap);
@@ -6399,11 +6400,6 @@ pmap_copy(pmap_t dst_pmap, pmap_t src_pmap, vm_offset_
PG_V = pmap_valid_bit(dst_pmap);
for (addr = src_addr; addr < end_addr; addr = va_next) {
- pt_entry_t *src_pte, *dst_pte;
- pml4_entry_t *pml4e;
- pdp_entry_t *pdpe;
- pd_entry_t srcptepaddr, *pde;
-
KASSERT(addr < UPT_MIN_ADDRESS,
("pmap_copy: invalid to pmap_copy page tables"));
@@ -6445,7 +6441,8 @@ pmap_copy(pmap_t dst_pmap, pmap_t src_pmap, vm_offset_
pmap_pv_insert_pde(dst_pmap, addr, srcptepaddr,
PMAP_ENTER_NORECLAIM, ))) {
*pde = srcptepaddr & ~PG_W;
- pmap_resident_count_inc(dst_pmap, NBPDR /
PAGE_SIZE);
+ pmap_resident_count_inc(dst_pmap, NBPDR /
+ PAGE_SIZE);
atomic_add_long(_pde_mappings, 1);
} else
dst_pdpg->wire_count--;
@@ -6463,58 +6460,54 @@ pmap_copy(pmap_t dst_pmap, pmap_t src_pmap, vm_offset_
src_pte = (pt_entry_t *)PHYS_TO_DMAP(srcptepaddr);
src_pte = _pte[pmap_pte_index(addr)];
dstmpte = NULL;
- while (addr < va_next) {
- pt_entry_t ptetemp;
+ for (; addr < va_next; addr += PAGE_SIZE, src_pte++) {
ptetemp = *src_pte;
+
/*
-* we only virtual copy managed pages
+* We only virtual copy managed pages.
*/
- if ((ptetemp & PG_MANAGED) != 0) {
- if (dstmpte != NULL &&
- dstmpte->pindex == pmap_pde_pindex(addr))
- dstmpte->wire_count++;
- else if ((dstmpte = pmap_allocpte(dst_pmap,
- addr, NULL)) == NULL)
- goto out;
- dst_pte = (pt_entry_t *)
- PHYS_TO_DMAP(VM_PAGE_TO_PHYS(dstmpte));
- dst_pte = _pte[pmap_pte_index(addr)];
- if (*dst_pte == 0 &&
- pmap_try_insert_pv_entry(dst_pmap, addr,
- PHYS_TO_VM_PAGE(ptetemp & PG_FRAME),
- )) {
+ if ((ptetemp & PG_MANAGED) == 0)
+ continue;
+
+ if (dstmpte != NULL) {
+ KASSERT(dstmpte->pindex ==
+ pmap_pde_pindex(addr),
+ ("dstmpte pindex/addr mismatch"));
+ dstmpte->wire_count++;
+ } else if ((dstmpte = pmap_allocpte(dst_pmap, addr,
+ NULL)) == NULL)
+ goto out;
+ dst_pte = (pt_entry_t *)
+
svn commit: r352519 - in stable/12/sys: amd64/amd64 arm64/arm64 i386/i386 riscv/riscv
Author: alc
Date: Thu Sep 19 15:12:32 2019
New Revision: 352519
URL: https://svnweb.freebsd.org/changeset/base/352519
Log:
MFC r350335:
Simplify the handling of superpages in pmap_clear_modify(). Specifically,
if a demotion succeeds, then all of the 4KB page mappings within the
superpage-sized region must be valid, so there is no point in testing the
validity of the 4KB page mapping that is going to be write protected.
Deindent the nearby code.
Modified:
stable/12/sys/amd64/amd64/pmap.c
stable/12/sys/arm64/arm64/pmap.c
stable/12/sys/i386/i386/pmap.c
stable/12/sys/riscv/riscv/pmap.c
Directory Properties:
stable/12/ (props changed)
Modified: stable/12/sys/amd64/amd64/pmap.c
==
--- stable/12/sys/amd64/amd64/pmap.cThu Sep 19 14:45:04 2019
(r352518)
+++ stable/12/sys/amd64/amd64/pmap.cThu Sep 19 15:12:32 2019
(r352519)
@@ -7599,7 +7599,7 @@ pmap_clear_modify(vm_page_t m)
pmap_t pmap;
pv_entry_t next_pv, pv;
pd_entry_t oldpde, *pde;
- pt_entry_t oldpte, *pte, PG_M, PG_RW, PG_V;
+ pt_entry_t *pte, PG_M, PG_RW;
struct rwlock *lock;
vm_offset_t va;
int md_gen, pvh_gen;
@@ -7635,33 +7635,23 @@ restart:
}
}
PG_M = pmap_modified_bit(pmap);
- PG_V = pmap_valid_bit(pmap);
PG_RW = pmap_rw_bit(pmap);
va = pv->pv_va;
pde = pmap_pde(pmap, va);
oldpde = *pde;
- if ((oldpde & PG_RW) != 0) {
- if (pmap_demote_pde_locked(pmap, pde, va, )) {
- if ((oldpde & PG_W) == 0) {
- /*
-* Write protect the mapping to a
-* single page so that a subsequent
-* write access may repromote.
-*/
- va += VM_PAGE_TO_PHYS(m) - (oldpde &
- PG_PS_FRAME);
- pte = pmap_pde_to_pte(pde, va);
- oldpte = *pte;
- if ((oldpte & PG_V) != 0) {
- while (!atomic_cmpset_long(pte,
- oldpte,
- oldpte & ~(PG_M | PG_RW)))
- oldpte = *pte;
- vm_page_dirty(m);
- pmap_invalidate_page(pmap, va);
- }
- }
- }
+ /* If oldpde has PG_RW set, then it also has PG_M set. */
+ if ((oldpde & PG_RW) != 0 &&
+ pmap_demote_pde_locked(pmap, pde, va, ) &&
+ (oldpde & PG_W) == 0) {
+ /*
+* Write protect the mapping to a single page so that
+* a subsequent write access may repromote.
+*/
+ va += VM_PAGE_TO_PHYS(m) - (oldpde & PG_PS_FRAME);
+ pte = pmap_pde_to_pte(pde, va);
+ atomic_clear_long(pte, PG_M | PG_RW);
+ vm_page_dirty(m);
+ pmap_invalidate_page(pmap, va);
}
PMAP_UNLOCK(pmap);
}
Modified: stable/12/sys/arm64/arm64/pmap.c
==
--- stable/12/sys/arm64/arm64/pmap.cThu Sep 19 14:45:04 2019
(r352518)
+++ stable/12/sys/arm64/arm64/pmap.cThu Sep 19 15:12:32 2019
(r352519)
@@ -5008,28 +5008,22 @@ restart:
va = pv->pv_va;
l2 = pmap_l2(pmap, va);
oldl2 = pmap_load(l2);
- if ((oldl2 & ATTR_SW_DBM) != 0) {
- if (pmap_demote_l2_locked(pmap, l2, va, )) {
- if ((oldl2 & ATTR_SW_WIRED) == 0) {
- /*
-* Write protect the mapping to a
-* single page so that a subsequent
-* write access may repromote.
-*/
- va += VM_PAGE_TO_PHYS(m) -
- (oldl2 & ~ATTR_MASK);
- l3 = pmap_l2_to_l3(l2, va);
- oldl3 = pmap_load(l3);
- if (pmap_l3_valid(oldl3)) {
-
svn commit: r352517 - in stable/12/sys: amd64/amd64 i386/i386
Author: alc
Date: Thu Sep 19 14:36:30 2019
New Revision: 352517
URL: https://svnweb.freebsd.org/changeset/base/352517
Log:
MFC r349526:
When we protect PTEs (as opposed to PDEs), we only call vm_page_dirty()
when, in fact, we are write protecting the page and the PTE has PG_M set.
However, pmap_protect_pde() was always calling vm_page_dirty() when the PDE
has PG_M set. So, adding PG_NX to a writeable PDE could result in
unnecessary (but harmless) calls to vm_page_dirty().
Simplify the loop calling vm_page_dirty() in pmap_protect_pde().
Modified:
stable/12/sys/amd64/amd64/pmap.c
stable/12/sys/i386/i386/pmap.c
Directory Properties:
stable/12/ (props changed)
Modified: stable/12/sys/amd64/amd64/pmap.c
==
--- stable/12/sys/amd64/amd64/pmap.cThu Sep 19 13:25:19 2019
(r352516)
+++ stable/12/sys/amd64/amd64/pmap.cThu Sep 19 14:36:30 2019
(r352517)
@@ -5214,8 +5214,7 @@ static boolean_t
pmap_protect_pde(pmap_t pmap, pd_entry_t *pde, vm_offset_t sva, vm_prot_t prot)
{
pd_entry_t newpde, oldpde;
- vm_offset_t eva, va;
- vm_page_t m;
+ vm_page_t m, mt;
boolean_t anychanged;
pt_entry_t PG_G, PG_M, PG_RW;
@@ -5229,15 +5228,15 @@ pmap_protect_pde(pmap_t pmap, pd_entry_t *pde, vm_offs
anychanged = FALSE;
retry:
oldpde = newpde = *pde;
- if ((oldpde & (PG_MANAGED | PG_M | PG_RW)) ==
- (PG_MANAGED | PG_M | PG_RW)) {
- eva = sva + NBPDR;
- for (va = sva, m = PHYS_TO_VM_PAGE(oldpde & PG_PS_FRAME);
- va < eva; va += PAGE_SIZE, m++)
- vm_page_dirty(m);
- }
- if ((prot & VM_PROT_WRITE) == 0)
+ if ((prot & VM_PROT_WRITE) == 0) {
+ if ((oldpde & (PG_MANAGED | PG_M | PG_RW)) ==
+ (PG_MANAGED | PG_M | PG_RW)) {
+ m = PHYS_TO_VM_PAGE(oldpde & PG_PS_FRAME);
+ for (mt = m; mt < [NBPDR / PAGE_SIZE]; mt++)
+ vm_page_dirty(mt);
+ }
newpde &= ~(PG_RW | PG_M);
+ }
if ((prot & VM_PROT_EXECUTE) == 0)
newpde |= pg_nx;
if (newpde != oldpde) {
Modified: stable/12/sys/i386/i386/pmap.c
==
--- stable/12/sys/i386/i386/pmap.c Thu Sep 19 13:25:19 2019
(r352516)
+++ stable/12/sys/i386/i386/pmap.c Thu Sep 19 14:36:30 2019
(r352517)
@@ -3332,8 +3332,7 @@ static boolean_t
pmap_protect_pde(pmap_t pmap, pd_entry_t *pde, vm_offset_t sva, vm_prot_t prot)
{
pd_entry_t newpde, oldpde;
- vm_offset_t eva, va;
- vm_page_t m;
+ vm_page_t m, mt;
boolean_t anychanged;
PMAP_LOCK_ASSERT(pmap, MA_OWNED);
@@ -3342,15 +3341,15 @@ pmap_protect_pde(pmap_t pmap, pd_entry_t *pde, vm_offs
anychanged = FALSE;
retry:
oldpde = newpde = *pde;
- if ((oldpde & (PG_MANAGED | PG_M | PG_RW)) ==
- (PG_MANAGED | PG_M | PG_RW)) {
- eva = sva + NBPDR;
- for (va = sva, m = PHYS_TO_VM_PAGE(oldpde & PG_PS_FRAME);
- va < eva; va += PAGE_SIZE, m++)
- vm_page_dirty(m);
- }
- if ((prot & VM_PROT_WRITE) == 0)
+ if ((prot & VM_PROT_WRITE) == 0) {
+ if ((oldpde & (PG_MANAGED | PG_M | PG_RW)) ==
+ (PG_MANAGED | PG_M | PG_RW)) {
+ m = PHYS_TO_VM_PAGE(oldpde & PG_PS_FRAME);
+ for (mt = m; mt < [NBPDR / PAGE_SIZE]; mt++)
+ vm_page_dirty(mt);
+ }
newpde &= ~(PG_RW | PG_M);
+ }
#if defined(PAE) || defined(PAE_TABLES)
if ((prot & VM_PROT_EXECUTE) == 0)
newpde |= pg_nx;
___
svn-src-all@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-all
To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"
svn commit: r352485 - stable/12/sys/arm64/arm64
Author: alc
Date: Wed Sep 18 14:38:42 2019
New Revision: 352485
URL: https://svnweb.freebsd.org/changeset/base/352485
Log:
MFC r349768
Restructure cache_handle_range to avoid repeated barriers. Specifically,
restructure cache_handle_range so that all of the data cache operations are
performed before any instruction cache operations. Then, we only need one
barrier between the data and instruction cache operations and one barrier
after the instruction cache operations.
On an Amazon EC2 a1.2xlarge instance, this simple change reduces the time
for a "make -j8 buildworld" by 9%.
Modified:
stable/12/sys/arm64/arm64/cpufunc_asm.S
Directory Properties:
stable/12/ (props changed)
Modified: stable/12/sys/arm64/arm64/cpufunc_asm.S
==
--- stable/12/sys/arm64/arm64/cpufunc_asm.S Wed Sep 18 14:27:09 2019
(r352484)
+++ stable/12/sys/arm64/arm64/cpufunc_asm.S Wed Sep 18 14:38:42 2019
(r352485)
@@ -47,7 +47,7 @@ __FBSDID("$FreeBSD$");
/*
* Macro to handle the cache. This takes the start address in x0, length
- * in x1. It will corrupt x0, x1, x2, and x3.
+ * in x1. It will corrupt x0, x1, x2, x3, and x4.
*/
.macro cache_handle_range dcop = 0, ic = 0, icop = 0
.if \ic == 0
@@ -60,17 +60,23 @@ __FBSDID("$FreeBSD$");
and x2, x0, x4 /* Get the low bits of the address */
add x1, x1, x2 /* Add these to the size */
bic x0, x0, x4 /* Clear the low bit of the address */
-1:
- dc \dcop, x0
- dsb ish
.if \ic != 0
- ic \icop, x0
- dsb ish
+ mov x2, x0 /* Save the address */
+ mov x4, x1 /* Save the size */
.endif
+1:
+ dc \dcop, x0
add x0, x0, x3 /* Move to the next line */
subsx1, x1, x3 /* Reduce the size */
b.hi1b /* Check if we are done */
+ dsb ish
.if \ic != 0
+2:
+ ic \icop, x2
+ add x2, x2, x3 /* Move to the next line */
+ subsx4, x4, x3 /* Reduce the size */
+ b.hi2b /* Check if we are done */
+ dsb ish
isb
.endif
.endm
___
svn-src-all@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-all
To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"
svn commit: r352484 - stable/12/sys/arm64/arm64
Author: alc
Date: Wed Sep 18 14:27:09 2019
New Revision: 352484
URL: https://svnweb.freebsd.org/changeset/base/352484
Log:
MFC r350546
Because of AArch64's weak memory consistency model, we need to include a
memory barrier between the stores for initializing a page table page and
the store for adding that page to the page table. Otherwise, a page table
walk by another processor's MMU could see the page table page before it
sees the initialized entries.
Simplify pmap_growkernel(). In particular, eliminate an unnecessary TLB
invalidation.
Modified:
stable/12/sys/arm64/arm64/pmap.c
Directory Properties:
stable/12/ (props changed)
Modified: stable/12/sys/arm64/arm64/pmap.c
==
--- stable/12/sys/arm64/arm64/pmap.cWed Sep 18 08:02:03 2019
(r352483)
+++ stable/12/sys/arm64/arm64/pmap.cWed Sep 18 14:27:09 2019
(r352484)
@@ -1525,6 +1525,16 @@ _pmap_alloc_l3(pmap_t pmap, vm_pindex_t ptepindex, str
pmap_zero_page(m);
/*
+* Because of AArch64's weak memory consistency model, we must have a
+* barrier here to ensure that the stores for zeroing "m", whether by
+* pmap_zero_page() or an earlier function, are visible before adding
+* "m" to the page table. Otherwise, a page table walk by another
+* processor's MMU could see the mapping to "m" and a stale, non-zero
+* PTE within "m".
+*/
+ dmb(ishst);
+
+ /*
* Map the pagetable page into the process address space, if
* it isn't already there.
*/
@@ -1775,12 +1785,14 @@ pmap_growkernel(vm_offset_t addr)
panic("pmap_growkernel: no memory to grow
kernel");
if ((nkpg->flags & PG_ZERO) == 0)
pmap_zero_page(nkpg);
+ /* See the dmb() in _pmap_alloc_l3(). */
+ dmb(ishst);
paddr = VM_PAGE_TO_PHYS(nkpg);
pmap_store(l1, paddr | L1_TABLE);
continue; /* try again */
}
l2 = pmap_l1_to_l2(l1, kernel_vm_end);
- if ((pmap_load(l2) & ATTR_AF) != 0) {
+ if (pmap_load(l2) != 0) {
kernel_vm_end = (kernel_vm_end + L2_SIZE) & ~L2_OFFSET;
if (kernel_vm_end - 1 >= vm_map_max(kernel_map)) {
kernel_vm_end = vm_map_max(kernel_map);
@@ -1796,9 +1808,10 @@ pmap_growkernel(vm_offset_t addr)
panic("pmap_growkernel: no memory to grow kernel");
if ((nkpg->flags & PG_ZERO) == 0)
pmap_zero_page(nkpg);
+ /* See the dmb() in _pmap_alloc_l3(). */
+ dmb(ishst);
paddr = VM_PAGE_TO_PHYS(nkpg);
- pmap_load_store(l2, paddr | L2_TABLE);
- pmap_invalidate_page(kernel_pmap, kernel_vm_end);
+ pmap_store(l2, paddr | L2_TABLE);
kernel_vm_end = (kernel_vm_end + L2_SIZE) & ~L2_OFFSET;
if (kernel_vm_end - 1 >= vm_map_max(kernel_map)) {
@@ -5537,6 +5550,10 @@ pmap_demote_l2_locked(pmap_t pmap, pt_entry_t *l2, vm_
/*
* If the page table page is not leftover from an earlier promotion,
* or the mapping attributes have changed, (re)initialize the L3 table.
+*
+* When pmap_update_entry() clears the old L2 mapping, it (indirectly)
+* performs a dsb(). That dsb() ensures that the stores for filling
+* "l3" are visible before "l3" is added to the page table.
*/
if (ml3->valid == 0 || (l3[0] & ATTR_MASK) != (newl3 & ATTR_MASK))
pmap_fill_l3(l3, newl3);
___
svn-src-all@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-all
To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"
svn commit: r352475 - in stable/12/sys: amd64/amd64 arm64/arm64 i386/i386
Author: alc
Date: Wed Sep 18 07:25:04 2019
New Revision: 352475
URL: https://svnweb.freebsd.org/changeset/base/352475
Log:
MFC r350463
In pmap_advise(), when we encounter a superpage mapping, we first demote
the mapping and then destroy one of the 4 KB page mappings so that there
is a potential trigger for repromotion. Currently, we destroy the first
4 KB page mapping that falls within the (current) superpage mapping or the
virtual address range [sva, eva). However, I have found empirically that
destroying the last 4 KB mapping produces slightly better results,
specifically, more promotions and fewer failed promotion attempts.
Accordingly, this revision changes pmap_advise() to destroy the last 4 KB
page mapping. It also replaces some nearby uses of boolean_t with bool.
Modified:
stable/12/sys/amd64/amd64/pmap.c
stable/12/sys/arm64/arm64/pmap.c
stable/12/sys/i386/i386/pmap.c
Directory Properties:
stable/12/ (props changed)
Modified: stable/12/sys/amd64/amd64/pmap.c
==
--- stable/12/sys/amd64/amd64/pmap.cWed Sep 18 07:24:33 2019
(r352474)
+++ stable/12/sys/amd64/amd64/pmap.cWed Sep 18 07:25:04 2019
(r352475)
@@ -7462,7 +7462,7 @@ pmap_advise(pmap_t pmap, vm_offset_t sva, vm_offset_t
pt_entry_t *pte, PG_A, PG_G, PG_M, PG_RW, PG_V;
vm_offset_t va, va_next;
vm_page_t m;
- boolean_t anychanged;
+ bool anychanged;
if (advice != MADV_DONTNEED && advice != MADV_FREE)
return;
@@ -7481,7 +7481,7 @@ pmap_advise(pmap_t pmap, vm_offset_t sva, vm_offset_t
PG_M = pmap_modified_bit(pmap);
PG_V = pmap_valid_bit(pmap);
PG_RW = pmap_rw_bit(pmap);
- anychanged = FALSE;
+ anychanged = false;
pmap_delayed_invl_start();
PMAP_LOCK(pmap);
for (; sva < eva; sva = va_next) {
@@ -7523,17 +7523,25 @@ pmap_advise(pmap_t pmap, vm_offset_t sva, vm_offset_t
/*
* Unless the page mappings are wired, remove the
* mapping to a single page so that a subsequent
-* access may repromote. Since the underlying page
-* table page is fully populated, this removal never
-* frees a page table page.
+* access may repromote. Choosing the last page
+* within the address range [sva, min(va_next, eva))
+* generally results in more repromotions. Since the
+* underlying page table page is fully populated, this
+* removal never frees a page table page.
*/
if ((oldpde & PG_W) == 0) {
- pte = pmap_pde_to_pte(pde, sva);
+ va = eva;
+ if (va > va_next)
+ va = va_next;
+ va -= PAGE_SIZE;
+ KASSERT(va >= sva,
+ ("pmap_advise: no address gap"));
+ pte = pmap_pde_to_pte(pde, va);
KASSERT((*pte & PG_V) != 0,
("pmap_advise: invalid PTE"));
- pmap_remove_pte(pmap, pte, sva, *pde, NULL,
+ pmap_remove_pte(pmap, pte, va, *pde, NULL,
);
- anychanged = TRUE;
+ anychanged = true;
}
if (lock != NULL)
rw_wunlock(lock);
@@ -7565,7 +7573,7 @@ pmap_advise(pmap_t pmap, vm_offset_t sva, vm_offset_t
if (va == va_next)
va = sva;
} else
- anychanged = TRUE;
+ anychanged = true;
continue;
maybe_invlrng:
if (va != va_next) {
Modified: stable/12/sys/arm64/arm64/pmap.c
==
--- stable/12/sys/arm64/arm64/pmap.cWed Sep 18 07:24:33 2019
(r352474)
+++ stable/12/sys/arm64/arm64/pmap.cWed Sep 18 07:25:04 2019
(r352475)
@@ -4882,15 +4882,23 @@ pmap_advise(pmap_t pmap, vm_offset_t sva, vm_offset_t
/*
* Unless the page mappings are wired, remove the
* mapping to a single page so that a subsequent
-* access may repromote. Since the underlying page
-* table page is fully populated, this removal never
-* frees a page table
svn commit: r352469 - stable/12/sys/arm64/arm64
Author: alc
Date: Wed Sep 18 07:01:01 2019
New Revision: 352469
URL: https://svnweb.freebsd.org/changeset/base/352469
Log:
MFC r350347
Implement pmap_advise(). (Without a working pmap_advise() implementation
madvise(MADV_DONTNEED) and madvise(MADV_FREE) are NOPs.)
Modified:
stable/12/sys/arm64/arm64/pmap.c
Directory Properties:
stable/12/ (props changed)
Modified: stable/12/sys/arm64/arm64/pmap.c
==
--- stable/12/sys/arm64/arm64/pmap.cWed Sep 18 06:50:29 2019
(r352468)
+++ stable/12/sys/arm64/arm64/pmap.cWed Sep 18 07:01:01 2019
(r352469)
@@ -2493,7 +2493,7 @@ pmap_remove_l2(pmap_t pmap, pt_entry_t *l2, vm_offset_
/*
* pmap_remove_l3: do the things to unmap a page in a process
*/
-static int __unused
+static int
pmap_remove_l3(pmap_t pmap, pt_entry_t *l3, vm_offset_t va,
pd_entry_t l2e, struct spglist *free, struct rwlock **lockp)
{
@@ -4833,6 +4833,110 @@ out:
void
pmap_advise(pmap_t pmap, vm_offset_t sva, vm_offset_t eva, int advice)
{
+ struct rwlock *lock;
+ vm_offset_t va, va_next;
+ vm_page_t m;
+ pd_entry_t *l0, *l1, *l2, oldl2;
+ pt_entry_t *l3, oldl3;
+
+ if (advice != MADV_DONTNEED && advice != MADV_FREE)
+ return;
+
+ PMAP_LOCK(pmap);
+ for (; sva < eva; sva = va_next) {
+ l0 = pmap_l0(pmap, sva);
+ if (pmap_load(l0) == 0) {
+ va_next = (sva + L0_SIZE) & ~L0_OFFSET;
+ if (va_next < sva)
+ va_next = eva;
+ continue;
+ }
+ l1 = pmap_l0_to_l1(l0, sva);
+ if (pmap_load(l1) == 0) {
+ va_next = (sva + L1_SIZE) & ~L1_OFFSET;
+ if (va_next < sva)
+ va_next = eva;
+ continue;
+ }
+ va_next = (sva + L2_SIZE) & ~L2_OFFSET;
+ if (va_next < sva)
+ va_next = eva;
+ l2 = pmap_l1_to_l2(l1, sva);
+ oldl2 = pmap_load(l2);
+ if (oldl2 == 0)
+ continue;
+ if ((oldl2 & ATTR_DESCR_MASK) == L2_BLOCK) {
+ if ((oldl2 & ATTR_SW_MANAGED) == 0)
+ continue;
+ lock = NULL;
+ if (!pmap_demote_l2_locked(pmap, l2, sva, )) {
+ if (lock != NULL)
+ rw_wunlock(lock);
+
+ /*
+* The 2MB page mapping was destroyed.
+*/
+ continue;
+ }
+
+ /*
+* Unless the page mappings are wired, remove the
+* mapping to a single page so that a subsequent
+* access may repromote. Since the underlying page
+* table page is fully populated, this removal never
+* frees a page table page.
+*/
+ if ((oldl2 & ATTR_SW_WIRED) == 0) {
+ l3 = pmap_l2_to_l3(l2, sva);
+ KASSERT(pmap_load(l3) != 0,
+ ("pmap_advise: invalid PTE"));
+ pmap_remove_l3(pmap, l3, sva, pmap_load(l2),
+ NULL, );
+ }
+ if (lock != NULL)
+ rw_wunlock(lock);
+ }
+ KASSERT((pmap_load(l2) & ATTR_DESCR_MASK) == L2_TABLE,
+ ("pmap_advise: invalid L2 entry after demotion"));
+ if (va_next > eva)
+ va_next = eva;
+ va = va_next;
+ for (l3 = pmap_l2_to_l3(l2, sva); sva != va_next; l3++,
+ sva += L3_SIZE) {
+ oldl3 = pmap_load(l3);
+ if ((oldl3 & (ATTR_SW_MANAGED | ATTR_DESCR_MASK)) !=
+ (ATTR_SW_MANAGED | L3_PAGE))
+ goto maybe_invlrng;
+ else if (pmap_pte_dirty(oldl3)) {
+ if (advice == MADV_DONTNEED) {
+ /*
+* Future calls to pmap_is_modified()
+* can be avoided by making the page
+* dirty now.
+*/
+ m = PHYS_TO_VM_PAGE(oldl3 & ~ATTR_MASK);
+ vm_page_dirty(m);
+ }
+ while
svn commit: r352452 - in stable/12/sys/arm64: arm64 include
Author: alc Date: Tue Sep 17 17:28:44 2019 New Revision: 352452 URL: https://svnweb.freebsd.org/changeset/base/352452 Log: MFC r349117, r349122, r349183, r349897, r349943, r350004, r350029, r350038, r350191, r350202, r350422, r350427, r350525 r349117: Three enhancements to arm64's pmap_protect(): Implement protection changes on superpage mappings. Previously, a superpage mapping was unconditionally demoted by pmap_protect(), even if the protection change applied to the entire superpage mapping. Precompute the bit mask describing the protection changes rather than recomputing it for every page table entry that is changed. Skip page table entries that already have the requested protection changes in place. r349122: Three changes to arm64's pmap_unwire(): Implement wiring changes on superpage mappings. Previously, a superpage mapping was unconditionally demoted by pmap_unwire(), even if the wiring change applied to the entire superpage mapping. Rewrite a comment to use the arm64 names for bits in a page table entry. Previously, the bits were referred to by their x86 names. Use atomic_"op"_64() instead of atomic_"op"_long() to update a page table entry in order to match the prevailing style in this file. r349183: Correct an error in r349122. pmap_unwire() should update the pmap's wired count, not its resident count. r349897: (by markj) Rename pmap_page_dirty() to pmap_pte_dirty(). This is a precursor to implementing dirty bit management. r349943: (by markj) Apply some light cleanup to uses of pmap_pte_dirty(). - Check for ATTR_SW_MANAGED before anything else. - Use pmap_pte_dirty() in pmap_remove_pages(). r350004: (by markj) Implement software access and dirty bit management for arm64. Previously the arm64 pmap did no reference or modification tracking; all mappings were treated as referenced and all read-write mappings were treated as dirty. This change implements software management of these attributes. Dirty bit management is implemented to emulate ARMv8.1's optional hardware dirty bit modifier management, following a suggestion from alc. In particular, a mapping with ATTR_SW_DBM set is logically writeable and is dirty if the ATTR_AP_RW_BIT bit is clear. Mappings with ATTR_AP_RW_BIT set are write-protected, and a write access will trigger a permission fault. pmap_fault() handles permission faults for such mappings and marks the page dirty by clearing ATTR_AP_RW_BIT, thus mapping the page read-write. r350029: (by markj) Propagate attribute changes during demotion. After r349117 and r349122, some mapping attribute changes do not trigger superpage demotion. However, pmap_demote_l2() was not updated to ensure that the replacement L3 entries carry any attribute changes that occurred since promotion. r350038: (by markj) Always use the software DBM bit for now. r350004 added most of the machinery needed to support hardware DBM management, but it did not intend to actually enable use of the hardware DBM bit. r350191: Introduce pmap_store(), and use it to replace pmap_load_store() in places where the page table entry was previously invalid. (Note that I did not replace pmap_load_store() when it was followed by a TLB invalidation, even if we are not using the return value from pmap_load_store().) Correct an error in pmap_enter(). A test for determining when to set PGA_WRITEABLE was always true, even if the mapping was read only. In pmap_enter_l2(), when replacing an empty kernel page table page by a superpage mapping, clear the old l2 entry and issue a TLB invalidation. My reading of the ARM architecture manual leads me to believe that the TLB could hold an intermediate entry referencing the empty kernel page table page even though it contains no valid mappings. Replace a couple direct uses of atomic_clear_64() by the new pmap_clear_bits(). In a couple comments, replace the term "paging-structure caches", which is an Intel-specific term for the caches that hold intermediate entries in the page table, with wording that is more consistent with the ARM architecture manual. r350202: With the introduction of software dirty bit emulation for managed mappings, we should test ATTR_SW_DBM, not ATTR_AP_RW, to determine whether to set PGA_WRITEABLE. In effect, we are currently setting PGA_WRITEABLE based on whether the dirty bit is preset, not whether the mapping is writeable. Correct this mistake. r350422: (by markj) Remove an unneeded trunc_page() in pmap_fault(). r350427: (by markj) Have arm64's pmap_fault() handle WnR faults on dirty PTEs. If we take a WnR permission fault on a managed, writeable and dirty PTE, simply
svn commit: r352374 - stable/12/sys/arm64/arm64
Author: alc
Date: Mon Sep 16 04:54:17 2019
New Revision: 352374
URL: https://svnweb.freebsd.org/changeset/base/352374
Log:
MFC r349323, r349442, r349866, r349975
pmap_enter_quick_locked() never replaces a valid mapping, so it need not
perform a TLB invalidation. A barrier suffices. (See r343876.)
Add a comment to pmap_enter_quick_locked() in order to highlight the fact
that it does not replace valid mappings.
Correct a typo in one of pmap_enter()'s comments.
Introduce pmap_clear(), which zeroes a page table entry, and use it,
instead of pmap_load_clear(), in places where we don't care about the page
table entry's prior contents.
Eliminate an unnecessary pmap_load() from pmap_remove_all(). Instead, use
the value returned by the pmap_load_clear() on the very next line.
A KASSERT() in pmap_enter(), which originated in the amd64 pmap, was meant
to check the value returned by the pmap_load_clear() on the previous
line. However, we were ignoring the value returned by the
pmap_load_clear(), and so the KASSERT() was not serving its intended
purpose. Use the value returned by the pmap_load_clear() in the
KASSERT().
Modified:
stable/12/sys/arm64/arm64/pmap.c
Directory Properties:
stable/12/ (props changed)
Modified: stable/12/sys/arm64/arm64/pmap.c
==
--- stable/12/sys/arm64/arm64/pmap.cMon Sep 16 02:31:58 2019
(r352373)
+++ stable/12/sys/arm64/arm64/pmap.cMon Sep 16 04:54:17 2019
(r352374)
@@ -315,6 +315,7 @@ static __inline vm_page_t pmap_remove_pt_page(pmap_t p
* They need to be atomic as the System MMU may write to the table at
* the same time as the CPU.
*/
+#definepmap_clear(table) atomic_store_64(table, 0)
#definepmap_load_store(table, entry) atomic_swap_64(table, entry)
#definepmap_set(table, mask) atomic_set_64(table, mask)
#definepmap_load_clear(table) atomic_swap_64(table, 0)
@@ -1208,7 +1209,7 @@ pmap_kremove(vm_offset_t va)
KASSERT(pte != NULL, ("pmap_kremove: Invalid address"));
KASSERT(lvl == 3, ("pmap_kremove: Invalid pte level %d", lvl));
- pmap_load_clear(pte);
+ pmap_clear(pte);
pmap_invalidate_page(kernel_pmap, va);
}
@@ -1230,7 +1231,7 @@ pmap_kremove_device(vm_offset_t sva, vm_size_t size)
KASSERT(pte != NULL, ("Invalid page table, va: 0x%lx", va));
KASSERT(lvl == 3,
("Invalid device pagetable level: %d != 3", lvl));
- pmap_load_clear(pte);
+ pmap_clear(pte);
va += PAGE_SIZE;
size -= PAGE_SIZE;
@@ -1315,7 +1316,7 @@ pmap_qremove(vm_offset_t sva, int count)
KASSERT(lvl == 3,
("Invalid device pagetable level: %d != 3", lvl));
if (pte != NULL) {
- pmap_load_clear(pte);
+ pmap_clear(pte);
}
va += PAGE_SIZE;
@@ -1374,19 +1375,19 @@ _pmap_unwire_l3(pmap_t pmap, vm_offset_t va, vm_page_t
pd_entry_t *l0;
l0 = pmap_l0(pmap, va);
- pmap_load_clear(l0);
+ pmap_clear(l0);
} else if (m->pindex >= NUL2E) {
/* l2 page */
pd_entry_t *l1;
l1 = pmap_l1(pmap, va);
- pmap_load_clear(l1);
+ pmap_clear(l1);
} else {
/* l3 page */
pd_entry_t *l2;
l2 = pmap_l2(pmap, va);
- pmap_load_clear(l2);
+ pmap_clear(l2);
}
pmap_resident_count_dec(pmap, 1);
if (m->pindex < NUL2E) {
@@ -2760,8 +2761,7 @@ retry:
tpde = pmap_load(pde);
pte = pmap_l2_to_l3(pde, pv->pv_va);
- tpte = pmap_load(pte);
- pmap_load_clear(pte);
+ tpte = pmap_load_clear(pte);
pmap_invalidate_page(pmap, pv->pv_va);
if (tpte & ATTR_SW_WIRED)
pmap->pm_stats.wired_count--;
@@ -2930,7 +2930,7 @@ pmap_update_entry(pmap_t pmap, pd_entry_t *pte, pd_ent
critical_enter();
/* Clear the old mapping */
- pmap_load_clear(pte);
+ pmap_clear(pte);
pmap_invalidate_range_nopin(pmap, va, va + size);
/* Create the new mapping */
@@ -3208,9 +3208,10 @@ havel3:
}
/*
-* The physical page has changed.
+* The physical page has changed. Temporarily invalidate
+* the mapping.
*/
- (void)pmap_load_clear(l3);
+ orig_l3 = pmap_load_clear(l3);
KASSERT((orig_l3 & ~ATTR_MASK) == opa,
("pmap_enter: unexpected pa update for %#lx", va));
if ((orig_l3 & ATTR_SW_MANAGED) != 0)
svn commit: r352373 - stable/12/sys/arm64/arm64
Author: alc
Date: Mon Sep 16 02:31:58 2019
New Revision: 352373
URL: https://svnweb.freebsd.org/changeset/base/352373
Log:
MFC r349003, r349031, r349042, r349129, r349290, r349618, r349798
Change pmap_demote_l2_locked() so that it removes the superpage mapping on
a demotion failure. Otherwise, some callers to pmap_demote_l2_locked(),
such as pmap_protect(), may leave an incorrect mapping in place on a
demotion failure.
Change pmap_demote_l2_locked() so that it handles addresses that are not
superpage aligned. Some callers to pmap_demote_l2_locked(), such as
pmap_protect(), may not pass a superpage aligned address.
Optimize TLB invalidation in pmap_remove_l2().
Change the arm64 pmap so that updates to the global count of wired pages
are not performed directly by the pmap. Instead, they are performed by
vm_page_free_pages_toq().
Batch the TLB invalidations that are performed by pmap_protect() rather
than performing them one at a time.
Eliminate a redundant call to pmap_invalidate_page() from
pmap_ts_referenced().
Introduce pmap_remove_l3_range() and use it in two places: (1)
pmap_remove(), where it eliminates redundant TLB invalidations by
pmap_remove() and pmap_remove_l3(), and (2) pmap_enter_l2(), where it may
optimize the TLB invalidations by batching them.
Implement pmap_copy().
Three changes to pmap_enter():
1. Use _pmap_alloc_l3() instead of pmap_alloc_l3() in order to handle the
possibility that a superpage mapping for "va" was created while we slept.
2. Eliminate code for allocating kernel page table pages. Kernel page
table pages are preallocated by pmap_growkernel().
3. Eliminate duplicated unlock operations when KERN_RESOURCE_SHORTAGE is
returned.
Modified:
stable/12/sys/arm64/arm64/pmap.c
Directory Properties:
stable/12/ (props changed)
Modified: stable/12/sys/arm64/arm64/pmap.c
==
--- stable/12/sys/arm64/arm64/pmap.cMon Sep 16 00:59:10 2019
(r352372)
+++ stable/12/sys/arm64/arm64/pmap.cMon Sep 16 02:31:58 2019
(r352373)
@@ -1410,8 +1410,6 @@ _pmap_unwire_l3(pmap_t pmap, vm_offset_t va, vm_page_t
}
pmap_invalidate_page(pmap, va);
- vm_wire_sub(1);
-
/*
* Put page on a list so that it is released after
* *ALL* TLB shootdown is done
@@ -2030,9 +2028,8 @@ next_chunk:
SLIST_REMOVE_HEAD(, plinks.s.ss);
/* Recycle a freed page table page. */
m_pc->wire_count = 1;
- vm_wire_add(1);
}
- vm_page_free_pages_toq(, false);
+ vm_page_free_pages_toq(, true);
return (m_pc);
}
@@ -2267,6 +2264,8 @@ pmap_pv_demote_l2(pmap_t pmap, vm_offset_t va, vm_padd
int bit, field;
PMAP_LOCK_ASSERT(pmap, MA_OWNED);
+ KASSERT((va & L2_OFFSET) == 0,
+ ("pmap_pv_demote_l2: va is not 2mpage aligned"));
KASSERT((pa & L2_OFFSET) == 0,
("pmap_pv_demote_l2: pa is not 2mpage aligned"));
CHANGE_PV_LIST_LOCK_TO_PHYS(lockp, pa);
@@ -2277,7 +2276,6 @@ pmap_pv_demote_l2(pmap_t pmap, vm_offset_t va, vm_padd
* must not be released until the last pv entry is reinstantiated.
*/
pvh = pa_to_pvh(pa);
- va = va & ~L2_OFFSET;
pv = pmap_pvh_remove(pvh, pmap, va);
KASSERT(pv != NULL, ("pmap_pv_demote_l2: pv not found"));
m = PHYS_TO_VM_PAGE(pa);
@@ -2433,7 +2431,13 @@ pmap_remove_l2(pmap_t pmap, pt_entry_t *l2, vm_offset_
old_l2 = pmap_load_clear(l2);
KASSERT((old_l2 & ATTR_DESCR_MASK) == L2_BLOCK,
("pmap_remove_l2: L2e %lx is not a block mapping", old_l2));
- pmap_invalidate_range(pmap, sva, sva + L2_SIZE);
+
+ /*
+* Since a promotion must break the 4KB page mappings before making
+* the 2MB page mapping, a pmap_invalidate_page() suffices.
+*/
+ pmap_invalidate_page(pmap, sva);
+
if (old_l2 & ATTR_SW_WIRED)
pmap->pm_stats.wired_count -= L2_SIZE / PAGE_SIZE;
pmap_resident_count_dec(pmap, L2_SIZE / PAGE_SIZE);
@@ -2463,8 +2467,7 @@ pmap_remove_l2(pmap_t pmap, pt_entry_t *l2, vm_offset_
pmap_resident_count_dec(pmap, 1);
KASSERT(ml3->wire_count == NL3PG,
("pmap_remove_l2: l3 page wire count error"));
- ml3->wire_count = 1;
- vm_page_unwire_noq(ml3);
+ ml3->wire_count = 0;
pmap_add_delayed_free_list(ml3, free, FALSE);
}
}
@@ -2507,6 +2510,82 @@ pmap_remove_l3(pmap_t pmap, pt_entry_t *l3, vm_offset_
}
/*
+ * Remove the specified range of addresses from the L3 page table that is
+ * identified by the given L2 entry.
+ */
+static void
svn commit: r352368 - stable/12/sys/arm64/arm64
Author: alc
Date: Sun Sep 15 21:32:19 2019
New Revision: 352368
URL: https://svnweb.freebsd.org/changeset/base/352368
Log:
MFC r349070
Previously, when pmap_remove_pages() destroyed a dirty superpage mapping,
it only called vm_page_dirty() on the first of the superpage's constituent
4KB pages. This revision corrects that error, calling vm_page_dirty() on
all of superpage's constituent 4KB pages.
Modified:
stable/12/sys/arm64/arm64/pmap.c
Directory Properties:
stable/12/ (props changed)
Modified: stable/12/sys/arm64/arm64/pmap.c
==
--- stable/12/sys/arm64/arm64/pmap.cSun Sep 15 21:27:14 2019
(r352367)
+++ stable/12/sys/arm64/arm64/pmap.cSun Sep 15 21:32:19 2019
(r352368)
@@ -4055,7 +4055,7 @@ pmap_remove_pages(pmap_t pmap)
switch (lvl) {
case 1:
for (mt = m; mt < [L2_SIZE /
PAGE_SIZE]; mt++)
- vm_page_dirty(m);
+ vm_page_dirty(mt);
break;
case 2:
vm_page_dirty(m);
___
svn-src-all@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-all
To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"
svn commit: r352367 - stable/12/sys/arm64/arm64
Author: alc
Date: Sun Sep 15 21:27:14 2019
New Revision: 352367
URL: https://svnweb.freebsd.org/changeset/base/352367
Log:
MFC r349905
According to Section D5.10.3 "Maintenance requirements on changing System
register values" of the architecture manual, an isb instruction should be
executed after updating ttbr0_el1 and before invalidating the TLB.
Modified:
stable/12/sys/arm64/arm64/efirt_machdep.c
stable/12/sys/arm64/arm64/pmap.c
Directory Properties:
stable/12/ (props changed)
Modified: stable/12/sys/arm64/arm64/efirt_machdep.c
==
--- stable/12/sys/arm64/arm64/efirt_machdep.c Sun Sep 15 21:13:13 2019
(r352366)
+++ stable/12/sys/arm64/arm64/efirt_machdep.c Sun Sep 15 21:27:14 2019
(r352367)
@@ -245,6 +245,7 @@ efi_arch_enter(void)
__asm __volatile(
"msr ttbr0_el1, %0 \n"
+ "isb\n"
"dsb ishst \n"
"tlbi vmalle1is \n"
"dsb ish \n"
@@ -262,6 +263,7 @@ efi_arch_leave(void)
td = curthread;
__asm __volatile(
"msr ttbr0_el1, %0 \n"
+ "isb\n"
"dsb ishst \n"
"tlbi vmalle1is \n"
"dsb ish \n"
Modified: stable/12/sys/arm64/arm64/pmap.c
==
--- stable/12/sys/arm64/arm64/pmap.cSun Sep 15 21:13:13 2019
(r352366)
+++ stable/12/sys/arm64/arm64/pmap.cSun Sep 15 21:27:14 2019
(r352367)
@@ -5175,8 +5175,10 @@ pmap_activate(struct thread *td)
critical_enter();
pmap = vmspace_pmap(td->td_proc->p_vmspace);
td->td_proc->p_md.md_l0addr = vtophys(pmap->pm_l0);
- __asm __volatile("msr ttbr0_el1, %0" : :
- "r"(td->td_proc->p_md.md_l0addr));
+ __asm __volatile(
+ "msr ttbr0_el1, %0 \n"
+ "isb\n"
+ : : "r"(td->td_proc->p_md.md_l0addr));
pmap_invalidate_all(pmap);
critical_exit();
}
___
svn-src-all@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-all
To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"
svn commit: r352358 - in stable/12/sys: amd64/amd64 arm64/arm64 i386/i386 riscv/riscv
Author: alc
Date: Sun Sep 15 17:22:29 2019
New Revision: 352358
URL: https://svnweb.freebsd.org/changeset/base/352358
Log:
MFC r348828
Implement an alternative solution to the amd64 and i386 pmap problem that
we previously addressed in r348246 (and MFCed in r348479).
This pmap problem also exists on arm64 and riscv. However, the original
solution developed for amd64 and i386 cannot be used on arm64 and riscv.
In particular, arm64 and riscv do not define a PG_PROMOTED flag in their
level 2 PTEs. (A PG_PROMOTED flag makes no sense on arm64, where unlike
x86 or riscv we are required to break the old 4KB mappings before making
the 2MB mapping; and on riscv there are no unused bits in the PTE to
define a PG_PROMOTED flag.)
This commit implements an alternative solution that can be used on all
four architectures. Moreover, this solution has two other advantages.
First, on older AMD processors that required the Erratum 383 workaround,
it is less costly. Specifically, it avoids unnecessary calls to
pmap_fill_ptp() on a superpage demotion. Second, it enables the
elimination of some calls to pagezero() in pmap_kernel_remove_{l2,pde}().
In addition, remove a related stale comment from pmap_enter_{l2,pde}().
Modified:
stable/12/sys/amd64/amd64/pmap.c
stable/12/sys/arm64/arm64/pmap.c
stable/12/sys/i386/i386/pmap.c
stable/12/sys/riscv/riscv/pmap.c
Directory Properties:
stable/12/ (props changed)
Modified: stable/12/sys/amd64/amd64/pmap.c
==
--- stable/12/sys/amd64/amd64/pmap.cSun Sep 15 16:53:02 2019
(r352357)
+++ stable/12/sys/amd64/amd64/pmap.cSun Sep 15 17:22:29 2019
(r352358)
@@ -1142,7 +1142,7 @@ static intpmap_enter_pde(pmap_t pmap, vm_offset_t
va,
static vm_page_t pmap_enter_quick_locked(pmap_t pmap, vm_offset_t va,
vm_page_t m, vm_prot_t prot, vm_page_t mpte, struct rwlock **lockp);
static void pmap_fill_ptp(pt_entry_t *firstpte, pt_entry_t newpte);
-static int pmap_insert_pt_page(pmap_t pmap, vm_page_t mpte);
+static int pmap_insert_pt_page(pmap_t pmap, vm_page_t mpte, bool promoted);
static void pmap_invalidate_cache_range_selfsnoop(vm_offset_t sva,
vm_offset_t eva);
static void pmap_invalidate_cache_range_all(vm_offset_t sva,
@@ -1846,8 +1846,13 @@ pmap_init(void)
mpte->pindex = pmap_pde_pindex(KERNBASE) + i;
mpte->phys_addr = KPTphys + (i << PAGE_SHIFT);
mpte->wire_count = 1;
+
+ /*
+* Collect the page table pages that were replaced by a 2MB
+* page in create_pagetables(). They are zero filled.
+*/
if (i << PDRSHIFT < KERNend &&
- pmap_insert_pt_page(kernel_pmap, mpte))
+ pmap_insert_pt_page(kernel_pmap, mpte, false))
panic("pmap_init: pmap_insert_pt_page failed");
}
PMAP_UNLOCK(kernel_pmap);
@@ -3219,12 +3224,15 @@ pmap_add_delayed_free_list(vm_page_t m, struct spglist
* of idle page table pages. Each of a pmap's page table pages is responsible
* for mapping a distinct range of virtual addresses. The pmap's collection is
* ordered by this virtual address range.
+ *
+ * If "promoted" is false, then the page table page "mpte" must be zero filled.
*/
static __inline int
-pmap_insert_pt_page(pmap_t pmap, vm_page_t mpte)
+pmap_insert_pt_page(pmap_t pmap, vm_page_t mpte, bool promoted)
{
PMAP_LOCK_ASSERT(pmap, MA_OWNED);
+ mpte->valid = promoted ? VM_PAGE_BITS_ALL : 0;
return (vm_radix_insert(>pm_root, mpte));
}
@@ -4716,7 +4724,7 @@ pmap_demote_pde_locked(pmap_t pmap, pd_entry_t *pde, v
* If the page table page is not leftover from an earlier promotion,
* initialize it.
*/
- if ((oldpde & PG_PROMOTED) == 0)
+ if (mpte->valid == 0)
pmap_fill_ptp(firstpte, newpte);
pmap_demote_pde_check(firstpte, newpte);
@@ -4789,9 +4797,11 @@ pmap_remove_kernel_pde(pmap_t pmap, pd_entry_t *pde, v
newpde = mptepa | X86_PG_M | X86_PG_A | X86_PG_RW | X86_PG_V;
/*
-* Initialize the page table page.
+* If this page table page was unmapped by a promotion, then it
+* contains valid mappings. Zero it to invalidate those mappings.
*/
- pagezero((void *)PHYS_TO_DMAP(mptepa));
+ if (mpte->valid != 0)
+ pagezero((void *)PHYS_TO_DMAP(mptepa));
/*
* Demote the mapping.
@@ -4856,6 +4866,8 @@ pmap_remove_pde(pmap_t pmap, pd_entry_t *pdq, vm_offse
} else {
mpte = pmap_remove_pt_page(pmap, sva);
if (mpte != NULL) {
+ KASSERT(mpte->valid == VM_PAGE_BITS_ALL,
+ ("pmap_remove_pde: pte page not promoted"));
svn commit: r350741 - in head/sys/arm64: arm64 include
Author: alc
Date: Thu Aug 8 06:26:34 2019
New Revision: 350741
URL: https://svnweb.freebsd.org/changeset/base/350741
Log:
Ordinarily, during a superpage promotion or demotion within a pmap, the
pmap's lock ensures that other operations on the pmap don't observe the
old mapping being broken before the new mapping is established. However,
pmap_kextract() doesn't acquire the kernel pmap's lock, so it may observe
the broken mapping. And, if it does, it returns an incorrect result.
This revision implements a lock-free solution to this problem in
pmap_update_entry() and pmap_kextract() because pmap_kextract() can't
acquire the kernel pmap's lock.
Reported by: andrew, greg_unrelenting.technology
Reviewed by: andrew, markj
Tested by:greg_unrelenting.technology
X-MFC with: r350579
Differential Revision:https://reviews.freebsd.org/D21169
Modified:
head/sys/arm64/arm64/pmap.c
head/sys/arm64/include/pte.h
Modified: head/sys/arm64/arm64/pmap.c
==
--- head/sys/arm64/arm64/pmap.c Thu Aug 8 04:29:46 2019(r350740)
+++ head/sys/arm64/arm64/pmap.c Thu Aug 8 06:26:34 2019(r350741)
@@ -1128,40 +1128,35 @@ vm_paddr_t
pmap_kextract(vm_offset_t va)
{
pt_entry_t *pte, tpte;
- vm_paddr_t pa;
- int lvl;
- if (va >= DMAP_MIN_ADDRESS && va < DMAP_MAX_ADDRESS) {
- pa = DMAP_TO_PHYS(va);
- } else {
- pa = 0;
- pte = pmap_pte(kernel_pmap, va, );
- if (pte != NULL) {
- tpte = pmap_load(pte);
- pa = tpte & ~ATTR_MASK;
- switch(lvl) {
- case 1:
- KASSERT((tpte & ATTR_DESCR_MASK) == L1_BLOCK,
- ("pmap_kextract: Invalid L1 pte found: %lx",
- tpte & ATTR_DESCR_MASK));
- pa |= (va & L1_OFFSET);
- break;
- case 2:
- KASSERT((tpte & ATTR_DESCR_MASK) == L2_BLOCK,
- ("pmap_kextract: Invalid L2 pte found: %lx",
- tpte & ATTR_DESCR_MASK));
- pa |= (va & L2_OFFSET);
- break;
- case 3:
- KASSERT((tpte & ATTR_DESCR_MASK) == L3_PAGE,
- ("pmap_kextract: Invalid L3 pte found: %lx",
- tpte & ATTR_DESCR_MASK));
- pa |= (va & L3_OFFSET);
- break;
- }
- }
- }
- return (pa);
+ if (va >= DMAP_MIN_ADDRESS && va < DMAP_MAX_ADDRESS)
+ return (DMAP_TO_PHYS(va));
+ pte = pmap_l1(kernel_pmap, va);
+ if (pte == NULL)
+ return (0);
+
+ /*
+* A concurrent pmap_update_entry() will clear the entry's valid bit
+* but leave the rest of the entry unchanged. Therefore, we treat a
+* non-zero entry as being valid, and we ignore the valid bit when
+* determining whether the entry maps a block, page, or table.
+*/
+ tpte = pmap_load(pte);
+ if (tpte == 0)
+ return (0);
+ if ((tpte & ATTR_DESCR_TYPE_MASK) == ATTR_DESCR_TYPE_BLOCK)
+ return ((tpte & ~ATTR_MASK) | (va & L1_OFFSET));
+ pte = pmap_l1_to_l2(, va);
+ tpte = pmap_load(pte);
+ if (tpte == 0)
+ return (0);
+ if ((tpte & ATTR_DESCR_TYPE_MASK) == ATTR_DESCR_TYPE_BLOCK)
+ return ((tpte & ~ATTR_MASK) | (va & L2_OFFSET));
+ pte = pmap_l2_to_l3(, va);
+ tpte = pmap_load(pte);
+ if (tpte == 0)
+ return (0);
+ return ((tpte & ~ATTR_MASK) | (va & L3_OFFSET));
}
/***
@@ -3021,8 +3016,12 @@ pmap_update_entry(pmap_t pmap, pd_entry_t *pte, pd_ent
intr = intr_disable();
critical_enter();
- /* Clear the old mapping */
- pmap_clear(pte);
+ /*
+* Clear the old mapping's valid bit, but leave the rest of the entry
+* unchanged, so that a lockless, concurrent pmap_kextract() can still
+* lookup the physical address.
+*/
+ pmap_clear_bits(pte, ATTR_DESCR_VALID);
pmap_invalidate_range_nopin(pmap, va, va + size);
/* Create the new mapping */
Modified: head/sys/arm64/include/pte.h
==
--- head/sys/arm64/include/pte.hThu Aug 8 04:29:46 2019
(r350740)
+++ head/sys/arm64/include/pte.hThu Aug 8 06:26:34 2019
(r350741)
@@ -71,7 +71,12 @@ typedef uint64_tpt_entry_t; /*
svn commit: r350579 - head/sys/arm64/arm64
Author: alc
Date: Mon Aug 5 02:44:04 2019
New Revision: 350579
URL: https://svnweb.freebsd.org/changeset/base/350579
Log:
Enable superpage promotion within the kernel pmap.
Reviewed by: markj
X-MFC after: r350004
Differential Revision:https://reviews.freebsd.org/D21149
Modified:
head/sys/arm64/arm64/pmap.c
Modified: head/sys/arm64/arm64/pmap.c
==
--- head/sys/arm64/arm64/pmap.c Mon Aug 5 01:37:18 2019(r350578)
+++ head/sys/arm64/arm64/pmap.c Mon Aug 5 02:44:04 2019(r350579)
@@ -3425,8 +3425,7 @@ validate:
}
#if VM_NRESERVLEVEL > 0
- if (pmap != pmap_kernel() &&
- (mpte == NULL || mpte->wire_count == NL3PG) &&
+ if ((mpte == NULL || mpte->wire_count == NL3PG) &&
pmap_ps_enabled(pmap) &&
(m->flags & PG_FICTITIOUS) == 0 &&
vm_reserv_level_iffullpop(m) == 0) {
___
svn-src-all@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-all
To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"
svn commit: r350546 - head/sys/arm64/arm64
Author: alc
Date: Fri Aug 2 22:36:42 2019
New Revision: 350546
URL: https://svnweb.freebsd.org/changeset/base/350546
Log:
Because of AArch64's weak memory consistency model, we need to include a
memory barrier between the stores for initializing a page table page and
the store for adding that page to the page table. Otherwise, a page table
walk by another processor's MMU could see the page table page before it
sees the initialized entries.
Simplify pmap_growkernel(). In particular, eliminate an unnecessary TLB
invalidation.
Reviewed by: andrew, markj
MFC after:1 week
Differential Revision:https://reviews.freebsd.org/D21126
Modified:
head/sys/arm64/arm64/pmap.c
Modified: head/sys/arm64/arm64/pmap.c
==
--- head/sys/arm64/arm64/pmap.c Fri Aug 2 22:33:43 2019(r350545)
+++ head/sys/arm64/arm64/pmap.c Fri Aug 2 22:36:42 2019(r350546)
@@ -1525,6 +1525,16 @@ _pmap_alloc_l3(pmap_t pmap, vm_pindex_t ptepindex, str
pmap_zero_page(m);
/*
+* Because of AArch64's weak memory consistency model, we must have a
+* barrier here to ensure that the stores for zeroing "m", whether by
+* pmap_zero_page() or an earlier function, are visible before adding
+* "m" to the page table. Otherwise, a page table walk by another
+* processor's MMU could see the mapping to "m" and a stale, non-zero
+* PTE within "m".
+*/
+ dmb(ishst);
+
+ /*
* Map the pagetable page into the process address space, if
* it isn't already there.
*/
@@ -1775,12 +1785,14 @@ pmap_growkernel(vm_offset_t addr)
panic("pmap_growkernel: no memory to grow
kernel");
if ((nkpg->flags & PG_ZERO) == 0)
pmap_zero_page(nkpg);
+ /* See the dmb() in _pmap_alloc_l3(). */
+ dmb(ishst);
paddr = VM_PAGE_TO_PHYS(nkpg);
pmap_store(l1, paddr | L1_TABLE);
continue; /* try again */
}
l2 = pmap_l1_to_l2(l1, kernel_vm_end);
- if ((pmap_load(l2) & ATTR_AF) != 0) {
+ if (pmap_load(l2) != 0) {
kernel_vm_end = (kernel_vm_end + L2_SIZE) & ~L2_OFFSET;
if (kernel_vm_end - 1 >= vm_map_max(kernel_map)) {
kernel_vm_end = vm_map_max(kernel_map);
@@ -1796,9 +1808,10 @@ pmap_growkernel(vm_offset_t addr)
panic("pmap_growkernel: no memory to grow kernel");
if ((nkpg->flags & PG_ZERO) == 0)
pmap_zero_page(nkpg);
+ /* See the dmb() in _pmap_alloc_l3(). */
+ dmb(ishst);
paddr = VM_PAGE_TO_PHYS(nkpg);
- pmap_load_store(l2, paddr | L2_TABLE);
- pmap_invalidate_page(kernel_pmap, kernel_vm_end);
+ pmap_store(l2, paddr | L2_TABLE);
kernel_vm_end = (kernel_vm_end + L2_SIZE) & ~L2_OFFSET;
if (kernel_vm_end - 1 >= vm_map_max(kernel_map)) {
@@ -5531,6 +5544,10 @@ pmap_demote_l2_locked(pmap_t pmap, pt_entry_t *l2, vm_
/*
* If the page table page is not leftover from an earlier promotion,
* or the mapping attributes have changed, (re)initialize the L3 table.
+*
+* When pmap_update_entry() clears the old L2 mapping, it (indirectly)
+* performs a dsb(). That dsb() ensures that the stores for filling
+* "l3" are visible before "l3" is added to the page table.
*/
if (ml3->valid == 0 || (l3[0] & ATTR_MASK) != (newl3 & ATTR_MASK))
pmap_fill_l3(l3, newl3);
___
svn-src-all@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-all
To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"
svn commit: r350463 - in head/sys: amd64/amd64 arm64/arm64 i386/i386
Author: alc
Date: Wed Jul 31 05:38:39 2019
New Revision: 350463
URL: https://svnweb.freebsd.org/changeset/base/350463
Log:
In pmap_advise(), when we encounter a superpage mapping, we first demote the
mapping and then destroy one of the 4 KB page mappings so that there is a
potential trigger for repromotion. Currently, we destroy the first 4 KB
page mapping that falls within the (current) superpage mapping or the
virtual address range [sva, eva). However, I have found empirically that
destroying the last 4 KB mapping produces slightly better results,
specifically, more promotions and fewer failed promotion attempts.
Accordingly, this revision changes pmap_advise() to destroy the last 4 KB
page mapping. It also replaces some nearby uses of boolean_t with bool.
Reviewed by: kib, markj
Differential Revision:https://reviews.freebsd.org/D21115
Modified:
head/sys/amd64/amd64/pmap.c
head/sys/arm64/arm64/pmap.c
head/sys/i386/i386/pmap.c
Modified: head/sys/amd64/amd64/pmap.c
==
--- head/sys/amd64/amd64/pmap.c Wed Jul 31 04:19:53 2019(r350462)
+++ head/sys/amd64/amd64/pmap.c Wed Jul 31 05:38:39 2019(r350463)
@@ -7444,7 +7444,7 @@ pmap_advise(pmap_t pmap, vm_offset_t sva, vm_offset_t
pt_entry_t *pte, PG_A, PG_G, PG_M, PG_RW, PG_V;
vm_offset_t va, va_next;
vm_page_t m;
- boolean_t anychanged;
+ bool anychanged;
if (advice != MADV_DONTNEED && advice != MADV_FREE)
return;
@@ -7463,7 +7463,7 @@ pmap_advise(pmap_t pmap, vm_offset_t sva, vm_offset_t
PG_M = pmap_modified_bit(pmap);
PG_V = pmap_valid_bit(pmap);
PG_RW = pmap_rw_bit(pmap);
- anychanged = FALSE;
+ anychanged = false;
pmap_delayed_invl_start();
PMAP_LOCK(pmap);
for (; sva < eva; sva = va_next) {
@@ -7505,17 +7505,25 @@ pmap_advise(pmap_t pmap, vm_offset_t sva, vm_offset_t
/*
* Unless the page mappings are wired, remove the
* mapping to a single page so that a subsequent
-* access may repromote. Since the underlying page
-* table page is fully populated, this removal never
-* frees a page table page.
+* access may repromote. Choosing the last page
+* within the address range [sva, min(va_next, eva))
+* generally results in more repromotions. Since the
+* underlying page table page is fully populated, this
+* removal never frees a page table page.
*/
if ((oldpde & PG_W) == 0) {
- pte = pmap_pde_to_pte(pde, sva);
+ va = eva;
+ if (va > va_next)
+ va = va_next;
+ va -= PAGE_SIZE;
+ KASSERT(va >= sva,
+ ("pmap_advise: no address gap"));
+ pte = pmap_pde_to_pte(pde, va);
KASSERT((*pte & PG_V) != 0,
("pmap_advise: invalid PTE"));
- pmap_remove_pte(pmap, pte, sva, *pde, NULL,
+ pmap_remove_pte(pmap, pte, va, *pde, NULL,
);
- anychanged = TRUE;
+ anychanged = true;
}
if (lock != NULL)
rw_wunlock(lock);
@@ -7547,7 +7555,7 @@ pmap_advise(pmap_t pmap, vm_offset_t sva, vm_offset_t
if (va == va_next)
va = sva;
} else
- anychanged = TRUE;
+ anychanged = true;
continue;
maybe_invlrng:
if (va != va_next) {
Modified: head/sys/arm64/arm64/pmap.c
==
--- head/sys/arm64/arm64/pmap.c Wed Jul 31 04:19:53 2019(r350462)
+++ head/sys/arm64/arm64/pmap.c Wed Jul 31 05:38:39 2019(r350463)
@@ -4888,15 +4888,23 @@ pmap_advise(pmap_t pmap, vm_offset_t sva, vm_offset_t
/*
* Unless the page mappings are wired, remove the
* mapping to a single page so that a subsequent
-* access may repromote. Since the underlying page
-* table page is fully populated, this removal never
-* frees a page table page.
+* access may
svn commit: r350347 - head/sys/arm64/arm64
Author: alc
Date: Fri Jul 26 05:07:09 2019
New Revision: 350347
URL: https://svnweb.freebsd.org/changeset/base/350347
Log:
Implement pmap_advise(). (Without a working pmap_advise() implementation
madvise(MADV_DONTNEED) and madvise(MADV_FREE) are NOPs.)
Reviewed by: markj
X-MFC after: r350004
Differential Revision:https://reviews.freebsd.org/D21062
Modified:
head/sys/arm64/arm64/pmap.c
Modified: head/sys/arm64/arm64/pmap.c
==
--- head/sys/arm64/arm64/pmap.c Fri Jul 26 03:49:16 2019(r350346)
+++ head/sys/arm64/arm64/pmap.c Fri Jul 26 05:07:09 2019(r350347)
@@ -2499,7 +2499,7 @@ pmap_remove_l2(pmap_t pmap, pt_entry_t *l2, vm_offset_
/*
* pmap_remove_l3: do the things to unmap a page in a process
*/
-static int __unused
+static int
pmap_remove_l3(pmap_t pmap, pt_entry_t *l3, vm_offset_t va,
pd_entry_t l2e, struct spglist *free, struct rwlock **lockp)
{
@@ -4839,6 +4839,110 @@ out:
void
pmap_advise(pmap_t pmap, vm_offset_t sva, vm_offset_t eva, int advice)
{
+ struct rwlock *lock;
+ vm_offset_t va, va_next;
+ vm_page_t m;
+ pd_entry_t *l0, *l1, *l2, oldl2;
+ pt_entry_t *l3, oldl3;
+
+ if (advice != MADV_DONTNEED && advice != MADV_FREE)
+ return;
+
+ PMAP_LOCK(pmap);
+ for (; sva < eva; sva = va_next) {
+ l0 = pmap_l0(pmap, sva);
+ if (pmap_load(l0) == 0) {
+ va_next = (sva + L0_SIZE) & ~L0_OFFSET;
+ if (va_next < sva)
+ va_next = eva;
+ continue;
+ }
+ l1 = pmap_l0_to_l1(l0, sva);
+ if (pmap_load(l1) == 0) {
+ va_next = (sva + L1_SIZE) & ~L1_OFFSET;
+ if (va_next < sva)
+ va_next = eva;
+ continue;
+ }
+ va_next = (sva + L2_SIZE) & ~L2_OFFSET;
+ if (va_next < sva)
+ va_next = eva;
+ l2 = pmap_l1_to_l2(l1, sva);
+ oldl2 = pmap_load(l2);
+ if (oldl2 == 0)
+ continue;
+ if ((oldl2 & ATTR_DESCR_MASK) == L2_BLOCK) {
+ if ((oldl2 & ATTR_SW_MANAGED) == 0)
+ continue;
+ lock = NULL;
+ if (!pmap_demote_l2_locked(pmap, l2, sva, )) {
+ if (lock != NULL)
+ rw_wunlock(lock);
+
+ /*
+* The 2MB page mapping was destroyed.
+*/
+ continue;
+ }
+
+ /*
+* Unless the page mappings are wired, remove the
+* mapping to a single page so that a subsequent
+* access may repromote. Since the underlying page
+* table page is fully populated, this removal never
+* frees a page table page.
+*/
+ if ((oldl2 & ATTR_SW_WIRED) == 0) {
+ l3 = pmap_l2_to_l3(l2, sva);
+ KASSERT(pmap_load(l3) != 0,
+ ("pmap_advise: invalid PTE"));
+ pmap_remove_l3(pmap, l3, sva, pmap_load(l2),
+ NULL, );
+ }
+ if (lock != NULL)
+ rw_wunlock(lock);
+ }
+ KASSERT((pmap_load(l2) & ATTR_DESCR_MASK) == L2_TABLE,
+ ("pmap_advise: invalid L2 entry after demotion"));
+ if (va_next > eva)
+ va_next = eva;
+ va = va_next;
+ for (l3 = pmap_l2_to_l3(l2, sva); sva != va_next; l3++,
+ sva += L3_SIZE) {
+ oldl3 = pmap_load(l3);
+ if ((oldl3 & (ATTR_SW_MANAGED | ATTR_DESCR_MASK)) !=
+ (ATTR_SW_MANAGED | L3_PAGE))
+ goto maybe_invlrng;
+ else if (pmap_pte_dirty(oldl3)) {
+ if (advice == MADV_DONTNEED) {
+ /*
+* Future calls to pmap_is_modified()
+* can be avoided by making the page
+* dirty now.
+*/
+ m = PHYS_TO_VM_PAGE(oldl3 & ~ATTR_MASK);
+ vm_page_dirty(m);
+ }
+
svn commit: r350335 - in head/sys: amd64/amd64 arm64/arm64 i386/i386 riscv/riscv
Author: alc
Date: Thu Jul 25 22:02:55 2019
New Revision: 350335
URL: https://svnweb.freebsd.org/changeset/base/350335
Log:
Simplify the handling of superpages in pmap_clear_modify(). Specifically,
if a demotion succeeds, then all of the 4KB page mappings within the
superpage-sized region must be valid, so there is no point in testing the
validity of the 4KB page mapping that is going to be write protected.
Deindent the nearby code.
Reviewed by: kib, markj
Tested by:pho (amd64, i386)
X-MFC after: r350004 (this change depends on arm64 dirty bit emulation)
Differential Revision:https://reviews.freebsd.org/D21027
Modified:
head/sys/amd64/amd64/pmap.c
head/sys/arm64/arm64/pmap.c
head/sys/i386/i386/pmap.c
head/sys/riscv/riscv/pmap.c
Modified: head/sys/amd64/amd64/pmap.c
==
--- head/sys/amd64/amd64/pmap.c Thu Jul 25 20:34:50 2019(r350334)
+++ head/sys/amd64/amd64/pmap.c Thu Jul 25 22:02:55 2019(r350335)
@@ -7574,7 +7574,7 @@ pmap_clear_modify(vm_page_t m)
pmap_t pmap;
pv_entry_t next_pv, pv;
pd_entry_t oldpde, *pde;
- pt_entry_t oldpte, *pte, PG_M, PG_RW, PG_V;
+ pt_entry_t *pte, PG_M, PG_RW;
struct rwlock *lock;
vm_offset_t va;
int md_gen, pvh_gen;
@@ -7610,33 +7610,23 @@ restart:
}
}
PG_M = pmap_modified_bit(pmap);
- PG_V = pmap_valid_bit(pmap);
PG_RW = pmap_rw_bit(pmap);
va = pv->pv_va;
pde = pmap_pde(pmap, va);
oldpde = *pde;
- if ((oldpde & PG_RW) != 0) {
- if (pmap_demote_pde_locked(pmap, pde, va, )) {
- if ((oldpde & PG_W) == 0) {
- /*
-* Write protect the mapping to a
-* single page so that a subsequent
-* write access may repromote.
-*/
- va += VM_PAGE_TO_PHYS(m) - (oldpde &
- PG_PS_FRAME);
- pte = pmap_pde_to_pte(pde, va);
- oldpte = *pte;
- if ((oldpte & PG_V) != 0) {
- while (!atomic_cmpset_long(pte,
- oldpte,
- oldpte & ~(PG_M | PG_RW)))
- oldpte = *pte;
- vm_page_dirty(m);
- pmap_invalidate_page(pmap, va);
- }
- }
- }
+ /* If oldpde has PG_RW set, then it also has PG_M set. */
+ if ((oldpde & PG_RW) != 0 &&
+ pmap_demote_pde_locked(pmap, pde, va, ) &&
+ (oldpde & PG_W) == 0) {
+ /*
+* Write protect the mapping to a single page so that
+* a subsequent write access may repromote.
+*/
+ va += VM_PAGE_TO_PHYS(m) - (oldpde & PG_PS_FRAME);
+ pte = pmap_pde_to_pte(pde, va);
+ atomic_clear_long(pte, PG_M | PG_RW);
+ vm_page_dirty(m);
+ pmap_invalidate_page(pmap, va);
}
PMAP_UNLOCK(pmap);
}
Modified: head/sys/arm64/arm64/pmap.c
==
--- head/sys/arm64/arm64/pmap.c Thu Jul 25 20:34:50 2019(r350334)
+++ head/sys/arm64/arm64/pmap.c Thu Jul 25 22:02:55 2019(r350335)
@@ -4889,28 +4889,22 @@ restart:
va = pv->pv_va;
l2 = pmap_l2(pmap, va);
oldl2 = pmap_load(l2);
- if ((oldl2 & ATTR_SW_DBM) != 0) {
- if (pmap_demote_l2_locked(pmap, l2, va, )) {
- if ((oldl2 & ATTR_SW_WIRED) == 0) {
- /*
-* Write protect the mapping to a
-* single page so that a subsequent
-* write access may repromote.
-*/
- va += VM_PAGE_TO_PHYS(m) -
- (oldl2 & ~ATTR_MASK);
- l3 = pmap_l2_to_l3(l2, va);
- oldl3 = pmap_load(l3);
-
svn commit: r350202 - head/sys/arm64/arm64
Author: alc Date: Sun Jul 21 17:00:19 2019 New Revision: 350202 URL: https://svnweb.freebsd.org/changeset/base/350202 Log: With the introduction of software dirty bit emulation for managed mappings, we should test ATTR_SW_DBM, not ATTR_AP_RW, to determine whether to set PGA_WRITEABLE. In effect, we are currently setting PGA_WRITEABLE based on whether the dirty bit is preset, not whether the mapping is writeable. Correct this mistake. Reviewed by: markj X-MFC with: r350004 Differential Revision:https://reviews.freebsd.org/D21013 Modified: head/sys/arm64/arm64/pmap.c Modified: head/sys/arm64/arm64/pmap.c == --- head/sys/arm64/arm64/pmap.c Sun Jul 21 16:35:49 2019(r350201) +++ head/sys/arm64/arm64/pmap.c Sun Jul 21 17:00:19 2019(r350202) @@ -3361,7 +3361,7 @@ havel3: CHANGE_PV_LIST_LOCK_TO_PHYS(, pa); TAILQ_INSERT_TAIL(>md.pv_list, pv, pv_next); m->md.pv_gen++; - if ((new_l3 & ATTR_AP_RW_BIT) == ATTR_AP(ATTR_AP_RW)) + if ((new_l3 & ATTR_SW_DBM) != 0) vm_page_aflag_set(m, PGA_WRITEABLE); } @@ -3554,7 +3554,7 @@ pmap_enter_l2(pmap_t pmap, vm_offset_t va, pd_entry_t va, pmap); return (KERN_RESOURCE_SHORTAGE); } - if ((new_l2 & ATTR_AP_RW_BIT) == ATTR_AP(ATTR_AP_RW)) + if ((new_l2 & ATTR_SW_DBM) != 0) for (mt = m; mt < [L2_SIZE / PAGE_SIZE]; mt++) vm_page_aflag_set(mt, PGA_WRITEABLE); } ___ svn-src-all@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"
svn commit: r350191 - head/sys/arm64/arm64
Author: alc
Date: Sun Jul 21 03:26:26 2019
New Revision: 350191
URL: https://svnweb.freebsd.org/changeset/base/350191
Log:
Introduce pmap_store(), and use it to replace pmap_load_store() in places
where the page table entry was previously invalid. (Note that I did not
replace pmap_load_store() when it was followed by a TLB invalidation, even
if we are not using the return value from pmap_load_store().)
Correct an error in pmap_enter(). A test for determining when to set
PGA_WRITEABLE was always true, even if the mapping was read only.
In pmap_enter_l2(), when replacing an empty kernel page table page by a
superpage mapping, clear the old l2 entry and issue a TLB invalidation. My
reading of the ARM architecture manual leads me to believe that the TLB
could hold an intermediate entry referencing the empty kernel page table
page even though it contains no valid mappings.
Replace a couple direct uses of atomic_clear_64() by the new
pmap_clear_bits().
In a couple comments, replace the term "paging-structure caches", which is
an Intel-specific term for the caches that hold intermediate entries in the
page table, with wording that is more consistent with the ARM architecture
manual.
Reviewed by: markj
X-MFC after: r350004
Differential Revision:https://reviews.freebsd.org/D20998
Modified:
head/sys/arm64/arm64/pmap.c
Modified: head/sys/arm64/arm64/pmap.c
==
--- head/sys/arm64/arm64/pmap.c Sun Jul 21 03:19:54 2019(r350190)
+++ head/sys/arm64/arm64/pmap.c Sun Jul 21 03:26:26 2019(r350191)
@@ -328,6 +328,7 @@ static __inline vm_page_t pmap_remove_pt_page(pmap_t p
#definepmap_load_clear(table) atomic_swap_64(table, 0)
#definepmap_load_store(table, entry) atomic_swap_64(table, entry)
#definepmap_set_bits(table, bits) atomic_set_64(table, bits)
+#definepmap_store(table, entry)atomic_store_64(table, entry)
//
/* Inline functions */
@@ -637,7 +638,7 @@ pmap_bootstrap_dmap(vm_offset_t kern_l1, vm_paddr_t mi
(vm_offset_t)l2);
freemempos += PAGE_SIZE;
- pmap_load_store(_dmap[l1_slot],
+ pmap_store(_dmap[l1_slot],
(l2_pa & ~Ln_TABLE_MASK) | L1_TABLE);
memset(l2, 0, PAGE_SIZE);
@@ -655,7 +656,7 @@ pmap_bootstrap_dmap(vm_offset_t kern_l1, vm_paddr_t mi
l2_slot = pmap_l2_index(va);
KASSERT(l2_slot != 0, ("..."));
- pmap_load_store([l2_slot],
+ pmap_store([l2_slot],
(pa & ~L2_OFFSET) | ATTR_DEFAULT | ATTR_XN |
ATTR_IDX(CACHED_MEMORY) | L2_BLOCK);
}
@@ -667,7 +668,7 @@ pmap_bootstrap_dmap(vm_offset_t kern_l1, vm_paddr_t mi
(physmap[i + 1] - pa) >= L1_SIZE;
pa += L1_SIZE, va += L1_SIZE) {
l1_slot = ((va - DMAP_MIN_ADDRESS) >> L1_SHIFT);
- pmap_load_store(_dmap[l1_slot],
+ pmap_store(_dmap[l1_slot],
(pa & ~L1_OFFSET) | ATTR_DEFAULT | ATTR_XN |
ATTR_IDX(CACHED_MEMORY) | L1_BLOCK);
}
@@ -682,7 +683,7 @@ pmap_bootstrap_dmap(vm_offset_t kern_l1, vm_paddr_t mi
(vm_offset_t)l2);
freemempos += PAGE_SIZE;
- pmap_load_store(_dmap[l1_slot],
+ pmap_store(_dmap[l1_slot],
(l2_pa & ~Ln_TABLE_MASK) | L1_TABLE);
memset(l2, 0, PAGE_SIZE);
@@ -692,7 +693,7 @@ pmap_bootstrap_dmap(vm_offset_t kern_l1, vm_paddr_t mi
for (; va < DMAP_MAX_ADDRESS && pa < physmap[i + 1];
pa += L2_SIZE, va += L2_SIZE) {
l2_slot = pmap_l2_index(va);
- pmap_load_store([l2_slot],
+ pmap_store([l2_slot],
(pa & ~L2_OFFSET) | ATTR_DEFAULT | ATTR_XN |
ATTR_IDX(CACHED_MEMORY) | L2_BLOCK);
}
@@ -727,7 +728,7 @@ pmap_bootstrap_l2(vm_offset_t l1pt, vm_offset_t va, vm
KASSERT(l1_slot < Ln_ENTRIES, ("Invalid L1 index"));
pa = pmap_early_vtophys(l1pt, l2pt);
- pmap_load_store([l1_slot],
+ pmap_store([l1_slot],
(pa & ~Ln_TABLE_MASK) | L1_TABLE);
l2pt += PAGE_SIZE;
}
@@ -757,7 +758,7 @@ pmap_bootstrap_l3(vm_offset_t
svn commit: r350021 - head/sys/i386/i386
Author: alc Date: Tue Jul 16 03:09:03 2019 New Revision: 350021 URL: https://svnweb.freebsd.org/changeset/base/350021 Log: Revert r349973. Upon further reflection, I realized that the comment deleted by r349973 is still valid on i386. Restore it. Discussed with: markj Modified: head/sys/i386/i386/pmap.c Modified: head/sys/i386/i386/pmap.c == --- head/sys/i386/i386/pmap.c Mon Jul 15 23:43:38 2019(r350020) +++ head/sys/i386/i386/pmap.c Tue Jul 16 03:09:03 2019(r350021) @@ -5096,6 +5096,13 @@ __CONCAT(PMTYPE, ts_referenced)(vm_page_t m) * reference bit will result in clearing that bit. * This function is designed to avoid the selection of * the same 4KB page for every 2- or 4MB page mapping. +* +* On demotion, a mapping that hasn't been referenced +* is simply destroyed. To avoid the possibility of a +* subsequent page fault on a demoted wired mapping, +* always leave its reference bit set. Moreover, +* since the superpage is wired, the current state of +* its reference bit won't affect page replacement. */ if pa >> PAGE_SHIFT) ^ (pv->pv_va >> PDRSHIFT) ^ (uintptr_t)pmap) & (NPTEPG - 1)) == 0 && ___ svn-src-all@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"
svn commit: r349975 - head/sys/arm64/arm64
Author: alc Date: Sat Jul 13 16:32:19 2019 New Revision: 349975 URL: https://svnweb.freebsd.org/changeset/base/349975 Log: Revert r349442, which was a workaround for bus errors caused by an errant TLB entry. Specifically, at the start of pmap_enter_quick_locked(), we would sometimes have a TLB entry for an invalid PTE, and we would need to issue a TLB invalidation before exiting pmap_enter_quick_locked(). However, we should never have a TLB entry for an invalid PTE. r349905 has addressed the root cause of the problem, and so we no longer need this workaround. X-MFC after: r349905 Modified: head/sys/arm64/arm64/pmap.c Modified: head/sys/arm64/arm64/pmap.c == --- head/sys/arm64/arm64/pmap.c Sat Jul 13 16:07:38 2019(r349974) +++ head/sys/arm64/arm64/pmap.c Sat Jul 13 16:32:19 2019(r349975) @@ -3713,14 +3713,7 @@ pmap_enter_quick_locked(pmap_t pmap, vm_offset_t va, v cpu_icache_sync_range(PHYS_TO_DMAP(pa), PAGE_SIZE); pmap_load_store(l3, l3_val); - - /* -* XXX In principle, because this L3 entry was invalid, we should not -* need to perform a TLB invalidation here. However, in practice, -* when simply performing a "dsb ishst" here, processes are being -* terminated due to bus errors and segmentation violations. -*/ - pmap_invalidate_page(pmap, va); + dsb(ishst); return (mpte); } ___ svn-src-all@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"
svn commit: r349973 - head/sys/i386/i386
Author: alc Date: Sat Jul 13 15:53:28 2019 New Revision: 349973 URL: https://svnweb.freebsd.org/changeset/base/349973 Log: Remove a stale comment. Reported by: markj MFC after:1 week Modified: head/sys/i386/i386/pmap.c Modified: head/sys/i386/i386/pmap.c == --- head/sys/i386/i386/pmap.c Sat Jul 13 15:34:29 2019(r349972) +++ head/sys/i386/i386/pmap.c Sat Jul 13 15:53:28 2019(r349973) @@ -5096,13 +5096,6 @@ __CONCAT(PMTYPE, ts_referenced)(vm_page_t m) * reference bit will result in clearing that bit. * This function is designed to avoid the selection of * the same 4KB page for every 2- or 4MB page mapping. -* -* On demotion, a mapping that hasn't been referenced -* is simply destroyed. To avoid the possibility of a -* subsequent page fault on a demoted wired mapping, -* always leave its reference bit set. Moreover, -* since the superpage is wired, the current state of -* its reference bit won't affect page replacement. */ if pa >> PAGE_SHIFT) ^ (pv->pv_va >> PDRSHIFT) ^ (uintptr_t)pmap) & (NPTEPG - 1)) == 0 && ___ svn-src-all@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"
svn commit: r349905 - head/sys/arm64/arm64
Author: alc
Date: Thu Jul 11 02:43:23 2019
New Revision: 349905
URL: https://svnweb.freebsd.org/changeset/base/349905
Log:
According to Section D5.10.3 "Maintenance requirements on changing System
register values" of the architecture manual, an isb instruction should be
executed after updating ttbr0_el1 and before invalidating the TLB. The
lack of this instruction in pmap_activate() appears to be the reason why
andrew@ and I have observed an unexpected TLB entry for an invalid PTE on
entry to pmap_enter_quick_locked(). Thus, we should now be able to revert
the workaround committed in r349442.
Reviewed by: markj
MFC after:1 week
Differential Revision:https://reviews.freebsd.org/D20904
Modified:
head/sys/arm64/arm64/efirt_machdep.c
head/sys/arm64/arm64/pmap.c
Modified: head/sys/arm64/arm64/efirt_machdep.c
==
--- head/sys/arm64/arm64/efirt_machdep.cThu Jul 11 02:15:50 2019
(r349904)
+++ head/sys/arm64/arm64/efirt_machdep.cThu Jul 11 02:43:23 2019
(r349905)
@@ -239,6 +239,7 @@ efi_arch_enter(void)
__asm __volatile(
"msr ttbr0_el1, %0 \n"
+ "isb\n"
"dsb ishst \n"
"tlbi vmalle1is \n"
"dsb ish \n"
@@ -266,6 +267,7 @@ efi_arch_leave(void)
td = curthread;
__asm __volatile(
"msr ttbr0_el1, %0 \n"
+ "isb\n"
"dsb ishst \n"
"tlbi vmalle1is \n"
"dsb ish \n"
Modified: head/sys/arm64/arm64/pmap.c
==
--- head/sys/arm64/arm64/pmap.c Thu Jul 11 02:15:50 2019(r349904)
+++ head/sys/arm64/arm64/pmap.c Thu Jul 11 02:43:23 2019(r349905)
@@ -5484,8 +5484,10 @@ pmap_activate(struct thread *td)
critical_enter();
pmap = vmspace_pmap(td->td_proc->p_vmspace);
td->td_proc->p_md.md_l0addr = vtophys(pmap->pm_l0);
- __asm __volatile("msr ttbr0_el1, %0" : :
- "r"(td->td_proc->p_md.md_l0addr));
+ __asm __volatile(
+ "msr ttbr0_el1, %0 \n"
+ "isb\n"
+ : : "r"(td->td_proc->p_md.md_l0addr));
pmap_invalidate_all(pmap);
critical_exit();
}
___
svn-src-all@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-all
To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"
svn commit: r349866 - head/sys/arm64/arm64
Author: alc
Date: Tue Jul 9 20:28:53 2019
New Revision: 349866
URL: https://svnweb.freebsd.org/changeset/base/349866
Log:
Introduce pmap_clear(), which zeroes a page table entry, and use it, instead
of pmap_load_clear(), in places where we don't care about the page table
entry's prior contents.
Eliminate an unnecessary pmap_load() from pmap_remove_all(). Instead, use
the value returned by the pmap_load_clear() on the very next line. (In the
future, when we support "hardware dirty bit management", using the value
from the pmap_load() rather than the pmap_load_clear() would have actually
been an error because the dirty bit could potentially change between the
pmap_load() and the pmap_load_clear().)
A KASSERT() in pmap_enter(), which originated in the amd64 pmap, was meant
to check the value returned by the pmap_load_clear() on the previous line.
However, we were ignoring the value returned by the pmap_load_clear(), and
so the KASSERT() was not serving its intended purpose. Use the value
returned by the pmap_load_clear() in the KASSERT().
MFC after:2 weeks
Modified:
head/sys/arm64/arm64/pmap.c
Modified: head/sys/arm64/arm64/pmap.c
==
--- head/sys/arm64/arm64/pmap.c Tue Jul 9 19:32:31 2019(r349865)
+++ head/sys/arm64/arm64/pmap.c Tue Jul 9 20:28:53 2019(r349866)
@@ -315,6 +315,7 @@ static __inline vm_page_t pmap_remove_pt_page(pmap_t p
* They need to be atomic as the System MMU may write to the table at
* the same time as the CPU.
*/
+#definepmap_clear(table) atomic_store_64(table, 0)
#definepmap_load_store(table, entry) atomic_swap_64(table, entry)
#definepmap_set(table, mask) atomic_set_64(table, mask)
#definepmap_load_clear(table) atomic_swap_64(table, 0)
@@ -1208,7 +1209,7 @@ pmap_kremove(vm_offset_t va)
KASSERT(pte != NULL, ("pmap_kremove: Invalid address"));
KASSERT(lvl == 3, ("pmap_kremove: Invalid pte level %d", lvl));
- pmap_load_clear(pte);
+ pmap_clear(pte);
pmap_invalidate_page(kernel_pmap, va);
}
@@ -1230,7 +1231,7 @@ pmap_kremove_device(vm_offset_t sva, vm_size_t size)
KASSERT(pte != NULL, ("Invalid page table, va: 0x%lx", va));
KASSERT(lvl == 3,
("Invalid device pagetable level: %d != 3", lvl));
- pmap_load_clear(pte);
+ pmap_clear(pte);
va += PAGE_SIZE;
size -= PAGE_SIZE;
@@ -1315,7 +1316,7 @@ pmap_qremove(vm_offset_t sva, int count)
KASSERT(lvl == 3,
("Invalid device pagetable level: %d != 3", lvl));
if (pte != NULL) {
- pmap_load_clear(pte);
+ pmap_clear(pte);
}
va += PAGE_SIZE;
@@ -1374,19 +1375,19 @@ _pmap_unwire_l3(pmap_t pmap, vm_offset_t va, vm_page_t
pd_entry_t *l0;
l0 = pmap_l0(pmap, va);
- pmap_load_clear(l0);
+ pmap_clear(l0);
} else if (m->pindex >= NUL2E) {
/* l2 page */
pd_entry_t *l1;
l1 = pmap_l1(pmap, va);
- pmap_load_clear(l1);
+ pmap_clear(l1);
} else {
/* l3 page */
pd_entry_t *l2;
l2 = pmap_l2(pmap, va);
- pmap_load_clear(l2);
+ pmap_clear(l2);
}
pmap_resident_count_dec(pmap, 1);
if (m->pindex < NUL2E) {
@@ -2760,8 +2761,7 @@ retry:
tpde = pmap_load(pde);
pte = pmap_l2_to_l3(pde, pv->pv_va);
- tpte = pmap_load(pte);
- pmap_load_clear(pte);
+ tpte = pmap_load_clear(pte);
pmap_invalidate_page(pmap, pv->pv_va);
if (tpte & ATTR_SW_WIRED)
pmap->pm_stats.wired_count--;
@@ -2986,7 +2986,7 @@ pmap_update_entry(pmap_t pmap, pd_entry_t *pte, pd_ent
critical_enter();
/* Clear the old mapping */
- pmap_load_clear(pte);
+ pmap_clear(pte);
pmap_invalidate_range_nopin(pmap, va, va + size);
/* Create the new mapping */
@@ -3265,9 +3265,10 @@ havel3:
}
/*
-* The physical page has changed.
+* The physical page has changed. Temporarily invalidate
+* the mapping.
*/
- (void)pmap_load_clear(l3);
+ orig_l3 = pmap_load_clear(l3);
KASSERT((orig_l3 & ~ATTR_MASK) == opa,
("pmap_enter: unexpected pa update for %#lx", va));
if ((orig_l3 & ATTR_SW_MANAGED) != 0) {
@@ -4282,7 +4283,12 @@ pmap_remove_pages(pmap_t pmap)
("pmap_remove_pages: bad pte %#jx",
svn commit: r349798 - head/sys/arm64/arm64
Author: alc
Date: Sun Jul 7 06:06:48 2019
New Revision: 349798
URL: https://svnweb.freebsd.org/changeset/base/349798
Log:
Three changes to pmap_enter():
1. Use _pmap_alloc_l3() instead of pmap_alloc_l3() in order to handle the
possibility that a superpage mapping for "va" was created while we slept.
(This is derived from the amd64 version.)
2. Eliminate code for allocating kernel page table pages. Kernel page
table pages are preallocated by pmap_growkernel().
3. Eliminate duplicated unlock operations when KERN_RESOURCE_SHORTAGE is
returned.
MFC after:2 weeks
Modified:
head/sys/arm64/arm64/pmap.c
Modified: head/sys/arm64/arm64/pmap.c
==
--- head/sys/arm64/arm64/pmap.c Sun Jul 7 05:30:07 2019(r349797)
+++ head/sys/arm64/arm64/pmap.c Sun Jul 7 06:06:48 2019(r349798)
@@ -3134,8 +3134,8 @@ pmap_enter(pmap_t pmap, vm_offset_t va, vm_page_t m, v
pt_entry_t new_l3, orig_l3;
pt_entry_t *l2, *l3;
pv_entry_t pv;
- vm_paddr_t opa, pa, l1_pa, l2_pa, l3_pa;
- vm_page_t mpte, om, l1_m, l2_m, l3_m;
+ vm_paddr_t opa, pa;
+ vm_page_t mpte, om;
boolean_t nosleep;
int lvl, rv;
@@ -3159,7 +3159,6 @@ pmap_enter(pmap_t pmap, vm_offset_t va, vm_page_t m, v
CTR2(KTR_PMAP, "pmap_enter: %.16lx -> %.16lx", va, pa);
lock = NULL;
- mpte = NULL;
PMAP_LOCK(pmap);
if (psind == 1) {
/* Assert the required virtual and physical alignment. */
@@ -3169,9 +3168,22 @@ pmap_enter(pmap_t pmap, vm_offset_t va, vm_page_t m, v
flags, m, );
goto out;
}
+ mpte = NULL;
+ /*
+* In the case that a page table page is not
+* resident, we are creating it here.
+*/
+retry:
pde = pmap_pde(pmap, va, );
- if (pde != NULL && lvl == 1) {
+ if (pde != NULL && lvl == 2) {
+ l3 = pmap_l2_to_l3(pde, va);
+ if (va < VM_MAXUSER_ADDRESS && mpte == NULL) {
+ mpte = PHYS_TO_VM_PAGE(pmap_load(pde) & ~ATTR_MASK);
+ mpte->wire_count++;
+ }
+ goto havel3;
+ } else if (pde != NULL && lvl == 1) {
l2 = pmap_l1_to_l2(pde, va);
if ((pmap_load(l2) & ATTR_DESCR_MASK) == L2_BLOCK &&
(l3 = pmap_demote_l2_locked(pmap, l2, va, )) != NULL) {
@@ -3183,83 +3195,26 @@ pmap_enter(pmap_t pmap, vm_offset_t va, vm_page_t m, v
}
goto havel3;
}
+ /* We need to allocate an L3 table. */
}
-
if (va < VM_MAXUSER_ADDRESS) {
nosleep = (flags & PMAP_ENTER_NOSLEEP) != 0;
- mpte = pmap_alloc_l3(pmap, va, nosleep ? NULL : );
+
+ /*
+* We use _pmap_alloc_l3() instead of pmap_alloc_l3() in order
+* to handle the possibility that a superpage mapping for "va"
+* was created while we slept.
+*/
+ mpte = _pmap_alloc_l3(pmap, pmap_l2_pindex(va),
+ nosleep ? NULL : );
if (mpte == NULL && nosleep) {
CTR0(KTR_PMAP, "pmap_enter: mpte == NULL");
- if (lock != NULL)
- rw_wunlock(lock);
- PMAP_UNLOCK(pmap);
- return (KERN_RESOURCE_SHORTAGE);
+ rv = KERN_RESOURCE_SHORTAGE;
+ goto out;
}
- pde = pmap_pde(pmap, va, );
- KASSERT(pde != NULL,
- ("pmap_enter: Invalid page entry, va: 0x%lx", va));
- KASSERT(lvl == 2,
- ("pmap_enter: Invalid level %d", lvl));
- } else {
- /*
-* If we get a level 2 pde it must point to a level 3 entry
-* otherwise we will need to create the intermediate tables
-*/
- if (lvl < 2) {
- switch (lvl) {
- default:
- case -1:
- /* Get the l0 pde to update */
- pde = pmap_l0(pmap, va);
- KASSERT(pde != NULL, ("..."));
-
- l1_m = vm_page_alloc(NULL, 0, VM_ALLOC_NORMAL |
- VM_ALLOC_NOOBJ | VM_ALLOC_WIRED |
- VM_ALLOC_ZERO);
- if (l1_m == NULL)
- panic("pmap_enter: l1 pte_m == NULL");
- if ((l1_m->flags & PG_ZERO) == 0)
- pmap_zero_page(l1_m);
-
- l1_pa = VM_PAGE_TO_PHYS(l1_m);
-
svn commit: r349768 - head/sys/arm64/arm64
Author: alc
Date: Fri Jul 5 20:01:06 2019
New Revision: 349768
URL: https://svnweb.freebsd.org/changeset/base/349768
Log:
Restructure cache_handle_range to avoid repeated barriers. Specifically,
restructure cache_handle_range so that all of the data cache operations are
performed before any instruction cache operations. Then, we only need one
barrier between the data and instruction cache operations and one barrier
after the instruction cache operations.
On an Amazon EC2 a1.2xlarge instance, this simple change reduces the time
for a "make -j8 buildworld" by 9%.
Reviewed by: andrew
MFC after:1 week
Differential Revision:https://reviews.freebsd.org/D20848
Modified:
head/sys/arm64/arm64/cpufunc_asm.S
Modified: head/sys/arm64/arm64/cpufunc_asm.S
==
--- head/sys/arm64/arm64/cpufunc_asm.S Fri Jul 5 16:49:34 2019
(r349767)
+++ head/sys/arm64/arm64/cpufunc_asm.S Fri Jul 5 20:01:06 2019
(r349768)
@@ -51,7 +51,7 @@ __FBSDID("$FreeBSD$");
/*
* Macro to handle the cache. This takes the start address in x0, length
- * in x1. It will corrupt x0, x1, x2, and x3.
+ * in x1. It will corrupt x0, x1, x2, x3, and x4.
*/
.macro cache_handle_range dcop = 0, ic = 0, icop = 0
.if \ic == 0
@@ -64,17 +64,23 @@ __FBSDID("$FreeBSD$");
and x2, x0, x4 /* Get the low bits of the address */
add x1, x1, x2 /* Add these to the size */
bic x0, x0, x4 /* Clear the low bit of the address */
-1:
- dc \dcop, x0
- dsb ish
.if \ic != 0
- ic \icop, x0
- dsb ish
+ mov x2, x0 /* Save the address */
+ mov x4, x1 /* Save the size */
.endif
+1:
+ dc \dcop, x0
add x0, x0, x3 /* Move to the next line */
subsx1, x1, x3 /* Reduce the size */
b.hi1b /* Check if we are done */
+ dsb ish
.if \ic != 0
+2:
+ ic \icop, x2
+ add x2, x2, x3 /* Move to the next line */
+ subsx4, x4, x3 /* Reduce the size */
+ b.hi2b /* Check if we are done */
+ dsb ish
isb
.endif
.endm
___
svn-src-all@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-all
To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"
svn commit: r349760 - head/sys/riscv/riscv
Author: alc
Date: Fri Jul 5 05:23:23 2019
New Revision: 349760
URL: https://svnweb.freebsd.org/changeset/base/349760
Log:
Merge r349526 from amd64. When we protect an L3 entry, we only call
vm_page_dirty() when, in fact, we are write protecting the page and the L3
entry has PTE_D set. However, pmap_protect() was always calling
vm_page_dirty() when an L2 entry has PTE_D set. Handle L2 entries the
same as L3 entries so that we won't perform unnecessary calls to
vm_page_dirty().
Simplify the loop calling vm_page_dirty() on L2 entries.
Modified:
head/sys/riscv/riscv/pmap.c
Modified: head/sys/riscv/riscv/pmap.c
==
--- head/sys/riscv/riscv/pmap.c Fri Jul 5 04:24:09 2019(r349759)
+++ head/sys/riscv/riscv/pmap.c Fri Jul 5 05:23:23 2019(r349760)
@@ -2298,9 +2298,9 @@ pmap_protect(pmap_t pmap, vm_offset_t sva, vm_offset_t
{
pd_entry_t *l1, *l2, l2e;
pt_entry_t *l3, l3e, mask;
- vm_page_t m;
+ vm_page_t m, mt;
vm_paddr_t pa;
- vm_offset_t va, va_next;
+ vm_offset_t va_next;
bool anychanged, pv_lists_locked;
if ((prot & VM_PROT_READ) == VM_PROT_NONE) {
@@ -2340,12 +2340,13 @@ resume:
if ((l2e & PTE_RWX) != 0) {
if (sva + L2_SIZE == va_next && eva >= va_next) {
retryl2:
- if ((l2e & (PTE_SW_MANAGED | PTE_D)) ==
+ if ((prot & VM_PROT_WRITE) == 0 &&
+ (l2e & (PTE_SW_MANAGED | PTE_D)) ==
(PTE_SW_MANAGED | PTE_D)) {
pa = PTE_TO_PHYS(l2e);
- for (va = sva, m = PHYS_TO_VM_PAGE(pa);
- va < va_next; m++, va += PAGE_SIZE)
- vm_page_dirty(m);
+ m = PHYS_TO_VM_PAGE(pa);
+ for (mt = m; mt < [Ln_ENTRIES]; mt++)
+ vm_page_dirty(mt);
}
if (!atomic_fcmpset_long(l2, , l2e & ~mask))
goto retryl2;
___
svn-src-all@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-all
To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"
svn commit: r349618 - head/sys/arm64/arm64
Author: alc
Date: Tue Jul 2 23:02:52 2019
New Revision: 349618
URL: https://svnweb.freebsd.org/changeset/base/349618
Log:
Implement pmap_copy(). (This includes the changes applied to the amd64
pmap_copy() in r349585.)
Reviewed by: kib, markj
Differential Revision:https://reviews.freebsd.org/D20790
Modified:
head/sys/arm64/arm64/pmap.c
Modified: head/sys/arm64/arm64/pmap.c
==
--- head/sys/arm64/arm64/pmap.c Tue Jul 2 22:58:21 2019(r349617)
+++ head/sys/arm64/arm64/pmap.c Tue Jul 2 23:02:52 2019(r349618)
@@ -3875,12 +3875,153 @@ pmap_unwire(pmap_t pmap, vm_offset_t sva, vm_offset_t
* in the destination map.
*
* This routine is only advisory and need not do anything.
+ *
+ * Because the executable mappings created by this routine are copied,
+ * it should not have to flush the instruction cache.
*/
-
void
pmap_copy(pmap_t dst_pmap, pmap_t src_pmap, vm_offset_t dst_addr, vm_size_t
len,
vm_offset_t src_addr)
{
+ struct rwlock *lock;
+ struct spglist free;
+ pd_entry_t *l0, *l1, *l2, srcptepaddr;
+ pt_entry_t *dst_pte, ptetemp, *src_pte;
+ vm_offset_t addr, end_addr, va_next;
+ vm_page_t dst_l2pg, dstmpte, srcmpte;
+
+ if (dst_addr != src_addr)
+ return;
+ end_addr = src_addr + len;
+ lock = NULL;
+ if (dst_pmap < src_pmap) {
+ PMAP_LOCK(dst_pmap);
+ PMAP_LOCK(src_pmap);
+ } else {
+ PMAP_LOCK(src_pmap);
+ PMAP_LOCK(dst_pmap);
+ }
+ for (addr = src_addr; addr < end_addr; addr = va_next) {
+ l0 = pmap_l0(src_pmap, addr);
+ if (pmap_load(l0) == 0) {
+ va_next = (addr + L0_SIZE) & ~L0_OFFSET;
+ if (va_next < addr)
+ va_next = end_addr;
+ continue;
+ }
+ l1 = pmap_l0_to_l1(l0, addr);
+ if (pmap_load(l1) == 0) {
+ va_next = (addr + L1_SIZE) & ~L1_OFFSET;
+ if (va_next < addr)
+ va_next = end_addr;
+ continue;
+ }
+ va_next = (addr + L2_SIZE) & ~L2_OFFSET;
+ if (va_next < addr)
+ va_next = end_addr;
+ l2 = pmap_l1_to_l2(l1, addr);
+ srcptepaddr = pmap_load(l2);
+ if (srcptepaddr == 0)
+ continue;
+ if ((srcptepaddr & ATTR_DESCR_MASK) == L2_BLOCK) {
+ if ((addr & L2_OFFSET) != 0 ||
+ addr + L2_SIZE > end_addr)
+ continue;
+ dst_l2pg = pmap_alloc_l2(dst_pmap, addr, NULL);
+ if (dst_l2pg == NULL)
+ break;
+ l2 = (pd_entry_t *)
+ PHYS_TO_DMAP(VM_PAGE_TO_PHYS(dst_l2pg));
+ l2 = [pmap_l2_index(addr)];
+ if (pmap_load(l2) == 0 &&
+ ((srcptepaddr & ATTR_SW_MANAGED) == 0 ||
+ pmap_pv_insert_l2(dst_pmap, addr, srcptepaddr,
+ PMAP_ENTER_NORECLAIM, ))) {
+ (void)pmap_load_store(l2, srcptepaddr &
+ ~ATTR_SW_WIRED);
+ pmap_resident_count_inc(dst_pmap, L2_SIZE /
+ PAGE_SIZE);
+ atomic_add_long(_l2_mappings, 1);
+ } else
+ dst_l2pg->wire_count--;
+ continue;
+ }
+ KASSERT((srcptepaddr & ATTR_DESCR_MASK) == L2_TABLE,
+ ("pmap_copy: invalid L2 entry"));
+ srcptepaddr &= ~ATTR_MASK;
+ srcmpte = PHYS_TO_VM_PAGE(srcptepaddr);
+ KASSERT(srcmpte->wire_count > 0,
+ ("pmap_copy: source page table page is unused"));
+ if (va_next > end_addr)
+ va_next = end_addr;
+ src_pte = (pt_entry_t *)PHYS_TO_DMAP(srcptepaddr);
+ src_pte = _pte[pmap_l3_index(addr)];
+ dstmpte = NULL;
+ for (; addr < va_next; addr += PAGE_SIZE, src_pte++) {
+ ptetemp = pmap_load(src_pte);
+
+ /*
+* We only virtual copy managed pages.
+*/
+ if ((ptetemp & ATTR_SW_MANAGED) == 0)
+ continue;
+
+ if (dstmpte != NULL) {
+ KASSERT(dstmpte->pindex == pmap_l2_pindex(addr),
+ ("dstmpte pindex/addr mismatch"));
+
svn commit: r349585 - head/sys/amd64/amd64
Author: alc
Date: Mon Jul 1 22:00:42 2019
New Revision: 349585
URL: https://svnweb.freebsd.org/changeset/base/349585
Log:
Tidy up pmap_copy(). Notably, deindent the innermost loop by making a
simple change to the control flow. Replace an unnecessary test by a
KASSERT. Add a comment explaining an obscure test.
Reviewed by: kib, markj
MFC after:3 weeks
Differential Revision:https://reviews.freebsd.org/D20812
Modified:
head/sys/amd64/amd64/pmap.c
Modified: head/sys/amd64/amd64/pmap.c
==
--- head/sys/amd64/amd64/pmap.c Mon Jul 1 21:50:53 2019(r349584)
+++ head/sys/amd64/amd64/pmap.c Mon Jul 1 22:00:42 2019(r349585)
@@ -6345,18 +6345,18 @@ pmap_unwire(pmap_t pmap, vm_offset_t sva, vm_offset_t
*
* This routine is only advisory and need not do anything.
*/
-
void
pmap_copy(pmap_t dst_pmap, pmap_t src_pmap, vm_offset_t dst_addr, vm_size_t
len,
vm_offset_t src_addr)
{
struct rwlock *lock;
struct spglist free;
- vm_offset_t addr;
- vm_offset_t end_addr = src_addr + len;
- vm_offset_t va_next;
+ pml4_entry_t *pml4e;
+ pdp_entry_t *pdpe;
+ pd_entry_t *pde, srcptepaddr;
+ pt_entry_t *dst_pte, PG_A, PG_M, PG_V, ptetemp, *src_pte;
+ vm_offset_t addr, end_addr, va_next;
vm_page_t dst_pdpg, dstmpte, srcmpte;
- pt_entry_t PG_A, PG_M, PG_V;
if (dst_addr != src_addr)
return;
@@ -6375,6 +6375,7 @@ pmap_copy(pmap_t dst_pmap, pmap_t src_pmap, vm_offset_
if (pmap_emulate_ad_bits(dst_pmap))
return;
+ end_addr = src_addr + len;
lock = NULL;
if (dst_pmap < src_pmap) {
PMAP_LOCK(dst_pmap);
@@ -6389,11 +6390,6 @@ pmap_copy(pmap_t dst_pmap, pmap_t src_pmap, vm_offset_
PG_V = pmap_valid_bit(dst_pmap);
for (addr = src_addr; addr < end_addr; addr = va_next) {
- pt_entry_t *src_pte, *dst_pte;
- pml4_entry_t *pml4e;
- pdp_entry_t *pdpe;
- pd_entry_t srcptepaddr, *pde;
-
KASSERT(addr < UPT_MIN_ADDRESS,
("pmap_copy: invalid to pmap_copy page tables"));
@@ -6435,7 +6431,8 @@ pmap_copy(pmap_t dst_pmap, pmap_t src_pmap, vm_offset_
pmap_pv_insert_pde(dst_pmap, addr, srcptepaddr,
PMAP_ENTER_NORECLAIM, ))) {
*pde = srcptepaddr & ~PG_W;
- pmap_resident_count_inc(dst_pmap, NBPDR /
PAGE_SIZE);
+ pmap_resident_count_inc(dst_pmap, NBPDR /
+ PAGE_SIZE);
atomic_add_long(_pde_mappings, 1);
} else
dst_pdpg->wire_count--;
@@ -6453,58 +6450,54 @@ pmap_copy(pmap_t dst_pmap, pmap_t src_pmap, vm_offset_
src_pte = (pt_entry_t *)PHYS_TO_DMAP(srcptepaddr);
src_pte = _pte[pmap_pte_index(addr)];
dstmpte = NULL;
- while (addr < va_next) {
- pt_entry_t ptetemp;
+ for (; addr < va_next; addr += PAGE_SIZE, src_pte++) {
ptetemp = *src_pte;
+
/*
-* we only virtual copy managed pages
+* We only virtual copy managed pages.
*/
- if ((ptetemp & PG_MANAGED) != 0) {
- if (dstmpte != NULL &&
- dstmpte->pindex == pmap_pde_pindex(addr))
- dstmpte->wire_count++;
- else if ((dstmpte = pmap_allocpte(dst_pmap,
- addr, NULL)) == NULL)
- goto out;
- dst_pte = (pt_entry_t *)
- PHYS_TO_DMAP(VM_PAGE_TO_PHYS(dstmpte));
- dst_pte = _pte[pmap_pte_index(addr)];
- if (*dst_pte == 0 &&
- pmap_try_insert_pv_entry(dst_pmap, addr,
- PHYS_TO_VM_PAGE(ptetemp & PG_FRAME),
- )) {
+ if ((ptetemp & PG_MANAGED) == 0)
+ continue;
+
+ if (dstmpte != NULL) {
+ KASSERT(dstmpte->pindex ==
+ pmap_pde_pindex(addr),
+ ("dstmpte pindex/addr mismatch"));
+ dstmpte->wire_count++;
+ } else if ((dstmpte = pmap_allocpte(dst_pmap, addr,
+ NULL)) == NULL)
+ goto out;
+ dst_pte =
svn commit: r349526 - in head/sys: amd64/amd64 i386/i386
Author: alc
Date: Fri Jun 28 22:40:34 2019
New Revision: 349526
URL: https://svnweb.freebsd.org/changeset/base/349526
Log:
When we protect PTEs (as opposed to PDEs), we only call vm_page_dirty()
when, in fact, we are write protecting the page and the PTE has PG_M set.
However, pmap_protect_pde() was always calling vm_page_dirty() when the PDE
has PG_M set. So, adding PG_NX to a writeable PDE could result in
unnecessary (but harmless) calls to vm_page_dirty().
Simplify the loop calling vm_page_dirty() in pmap_protect_pde().
Reviewed by: kib, markj
MFC after:1 week
Differential Revision:https://reviews.freebsd.org/D20793
Modified:
head/sys/amd64/amd64/pmap.c
head/sys/i386/i386/pmap.c
Modified: head/sys/amd64/amd64/pmap.c
==
--- head/sys/amd64/amd64/pmap.c Fri Jun 28 22:36:27 2019(r349525)
+++ head/sys/amd64/amd64/pmap.c Fri Jun 28 22:40:34 2019(r349526)
@@ -5202,8 +5202,7 @@ static boolean_t
pmap_protect_pde(pmap_t pmap, pd_entry_t *pde, vm_offset_t sva, vm_prot_t prot)
{
pd_entry_t newpde, oldpde;
- vm_offset_t eva, va;
- vm_page_t m;
+ vm_page_t m, mt;
boolean_t anychanged;
pt_entry_t PG_G, PG_M, PG_RW;
@@ -5217,15 +5216,15 @@ pmap_protect_pde(pmap_t pmap, pd_entry_t *pde, vm_offs
anychanged = FALSE;
retry:
oldpde = newpde = *pde;
- if ((oldpde & (PG_MANAGED | PG_M | PG_RW)) ==
- (PG_MANAGED | PG_M | PG_RW)) {
- eva = sva + NBPDR;
- for (va = sva, m = PHYS_TO_VM_PAGE(oldpde & PG_PS_FRAME);
- va < eva; va += PAGE_SIZE, m++)
- vm_page_dirty(m);
- }
- if ((prot & VM_PROT_WRITE) == 0)
+ if ((prot & VM_PROT_WRITE) == 0) {
+ if ((oldpde & (PG_MANAGED | PG_M | PG_RW)) ==
+ (PG_MANAGED | PG_M | PG_RW)) {
+ m = PHYS_TO_VM_PAGE(oldpde & PG_PS_FRAME);
+ for (mt = m; mt < [NBPDR / PAGE_SIZE]; mt++)
+ vm_page_dirty(mt);
+ }
newpde &= ~(PG_RW | PG_M);
+ }
if ((prot & VM_PROT_EXECUTE) == 0)
newpde |= pg_nx;
if (newpde != oldpde) {
Modified: head/sys/i386/i386/pmap.c
==
--- head/sys/i386/i386/pmap.c Fri Jun 28 22:36:27 2019(r349525)
+++ head/sys/i386/i386/pmap.c Fri Jun 28 22:40:34 2019(r349526)
@@ -3251,8 +3251,7 @@ static boolean_t
pmap_protect_pde(pmap_t pmap, pd_entry_t *pde, vm_offset_t sva, vm_prot_t prot)
{
pd_entry_t newpde, oldpde;
- vm_offset_t eva, va;
- vm_page_t m;
+ vm_page_t m, mt;
boolean_t anychanged;
PMAP_LOCK_ASSERT(pmap, MA_OWNED);
@@ -3261,15 +3260,15 @@ pmap_protect_pde(pmap_t pmap, pd_entry_t *pde, vm_offs
anychanged = FALSE;
retry:
oldpde = newpde = *pde;
- if ((oldpde & (PG_MANAGED | PG_M | PG_RW)) ==
- (PG_MANAGED | PG_M | PG_RW)) {
- eva = sva + NBPDR;
- for (va = sva, m = PHYS_TO_VM_PAGE(oldpde & PG_PS_FRAME);
- va < eva; va += PAGE_SIZE, m++)
- vm_page_dirty(m);
- }
- if ((prot & VM_PROT_WRITE) == 0)
+ if ((prot & VM_PROT_WRITE) == 0) {
+ if ((oldpde & (PG_MANAGED | PG_M | PG_RW)) ==
+ (PG_MANAGED | PG_M | PG_RW)) {
+ m = PHYS_TO_VM_PAGE(oldpde & PG_PS_FRAME);
+ for (mt = m; mt < [NBPDR / PAGE_SIZE]; mt++)
+ vm_page_dirty(mt);
+ }
newpde &= ~(PG_RW | PG_M);
+ }
#ifdef PMAP_PAE_COMP
if ((prot & VM_PROT_EXECUTE) == 0 && !i386_read_exec)
newpde |= pg_nx;
___
svn-src-all@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-all
To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"
svn commit: r349442 - head/sys/arm64/arm64
Author: alc Date: Wed Jun 26 21:43:41 2019 New Revision: 349442 URL: https://svnweb.freebsd.org/changeset/base/349442 Log: Revert one of the changes from r349323. Specifically, undo the change that replaced a pmap_invalidate_page() with a dsb(ishst) in pmap_enter_quick_locked(). Even though this change is in principle correct, I am seeing occasional, spurious bus errors that are only reproducible without this pmap_invalidate_page(). (None of adding an isb, "upgrading" the dsb to wait on loads as well as stores, or disabling superpage mappings eliminates the bus errors.) Add an XXX comment explaining why the pmap_invalidate_page() is being performed. Discussed with:andrew, markj Modified: head/sys/arm64/arm64/pmap.c Modified: head/sys/arm64/arm64/pmap.c == --- head/sys/arm64/arm64/pmap.c Wed Jun 26 21:19:43 2019(r349441) +++ head/sys/arm64/arm64/pmap.c Wed Jun 26 21:43:41 2019(r349442) @@ -3758,7 +3758,15 @@ pmap_enter_quick_locked(pmap_t pmap, vm_offset_t va, v cpu_icache_sync_range(PHYS_TO_DMAP(pa), PAGE_SIZE); pmap_load_store(l3, l3_val); - dsb(ishst); + + /* +* XXX In principle, because this L3 entry was invalid, we should not +* need to perform a TLB invalidation here. However, in practice, +* when simply performing a "dsb ishst" here, processes are being +* terminated due to bus errors and segmentation violations. +*/ + pmap_invalidate_page(pmap, va); + return (mpte); } ___ svn-src-all@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"
svn commit: r349323 - head/sys/arm64/arm64
Author: alc
Date: Sun Jun 23 21:06:56 2019
New Revision: 349323
URL: https://svnweb.freebsd.org/changeset/base/349323
Log:
pmap_enter_quick_locked() never replaces a valid mapping, so it need not
perform a TLB invalidation. A barrier suffices. (See r343876.)
Add a comment to pmap_enter_quick_locked() in order to highlight the
fact that it does not replace valid mappings.
Correct a typo in one of pmap_enter()'s comments.
MFC after:1 week
Modified:
head/sys/arm64/arm64/pmap.c
Modified: head/sys/arm64/arm64/pmap.c
==
--- head/sys/arm64/arm64/pmap.c Sun Jun 23 19:37:12 2019(r349322)
+++ head/sys/arm64/arm64/pmap.c Sun Jun 23 21:06:56 2019(r349323)
@@ -3408,7 +3408,7 @@ validate:
__func__, pmap, va, new_l3);
}
} else {
- /* New mappig */
+ /* New mapping */
pmap_load_store(l3, new_l3);
dsb(ishst);
}
@@ -3706,6 +3706,9 @@ pmap_enter_quick_locked(pmap_t pmap, vm_offset_t va, v
l3 = pmap_l2_to_l3(pde, va);
}
+ /*
+* Abort if a mapping already exists.
+*/
if (pmap_load(l3) != 0) {
if (mpte != NULL) {
mpte->wire_count--;
@@ -3755,7 +3758,7 @@ pmap_enter_quick_locked(pmap_t pmap, vm_offset_t va, v
cpu_icache_sync_range(PHYS_TO_DMAP(pa), PAGE_SIZE);
pmap_load_store(l3, l3_val);
- pmap_invalidate_page(pmap, va);
+ dsb(ishst);
return (mpte);
}
___
svn-src-all@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-all
To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"
svn commit: r349290 - head/sys/arm64/arm64
Author: alc
Date: Sat Jun 22 16:26:38 2019
New Revision: 349290
URL: https://svnweb.freebsd.org/changeset/base/349290
Log:
Introduce pmap_remove_l3_range() and use it in two places:
(1) pmap_remove(), where it eliminates redundant TLB invalidations by
pmap_remove() and pmap_remove_l3(), and (2) pmap_enter_l2(), where it may
optimize the TLB invalidations by batching them.
Reviewed by: markj
MFC after:1 week
Differential Revision:https://reviews.freebsd.org/D12725
Modified:
head/sys/arm64/arm64/pmap.c
Modified: head/sys/arm64/arm64/pmap.c
==
--- head/sys/arm64/arm64/pmap.c Sat Jun 22 15:39:34 2019(r349289)
+++ head/sys/arm64/arm64/pmap.c Sat Jun 22 16:26:38 2019(r349290)
@@ -2510,6 +2510,82 @@ pmap_remove_l3(pmap_t pmap, pt_entry_t *l3, vm_offset_
}
/*
+ * Remove the specified range of addresses from the L3 page table that is
+ * identified by the given L2 entry.
+ */
+static void
+pmap_remove_l3_range(pmap_t pmap, pd_entry_t l2e, vm_offset_t sva,
+vm_offset_t eva, struct spglist *free, struct rwlock **lockp)
+{
+ struct md_page *pvh;
+ struct rwlock *new_lock;
+ pt_entry_t *l3, old_l3;
+ vm_offset_t va;
+ vm_page_t m;
+
+ PMAP_LOCK_ASSERT(pmap, MA_OWNED);
+ KASSERT(rounddown2(sva, L2_SIZE) + L2_SIZE == roundup2(eva, L2_SIZE),
+ ("pmap_remove_l3_range: range crosses an L3 page table boundary"));
+ va = eva;
+ for (l3 = pmap_l2_to_l3(, sva); sva != eva; l3++, sva += L3_SIZE) {
+ if (!pmap_l3_valid(pmap_load(l3))) {
+ if (va != eva) {
+ pmap_invalidate_range(pmap, va, sva);
+ va = eva;
+ }
+ continue;
+ }
+ old_l3 = pmap_load_clear(l3);
+ if ((old_l3 & ATTR_SW_WIRED) != 0)
+ pmap->pm_stats.wired_count--;
+ pmap_resident_count_dec(pmap, 1);
+ if ((old_l3 & ATTR_SW_MANAGED) != 0) {
+ m = PHYS_TO_VM_PAGE(old_l3 & ~ATTR_MASK);
+ if (pmap_page_dirty(old_l3))
+ vm_page_dirty(m);
+ if ((old_l3 & ATTR_AF) != 0)
+ vm_page_aflag_set(m, PGA_REFERENCED);
+ new_lock = PHYS_TO_PV_LIST_LOCK(VM_PAGE_TO_PHYS(m));
+ if (new_lock != *lockp) {
+ if (*lockp != NULL) {
+ /*
+* Pending TLB invalidations must be
+* performed before the PV list lock is
+* released. Otherwise, a concurrent
+* pmap_remove_all() on a physical page
+* could return while a stale TLB entry
+* still provides access to that page.
+*/
+ if (va != eva) {
+ pmap_invalidate_range(pmap, va,
+ sva);
+ va = eva;
+ }
+ rw_wunlock(*lockp);
+ }
+ *lockp = new_lock;
+ rw_wlock(*lockp);
+ }
+ pmap_pvh_free(>md, pmap, sva);
+ if (TAILQ_EMPTY(>md.pv_list) &&
+ (m->flags & PG_FICTITIOUS) == 0) {
+ pvh = pa_to_pvh(VM_PAGE_TO_PHYS(m));
+ if (TAILQ_EMPTY(>pv_list))
+ vm_page_aflag_clear(m, PGA_WRITEABLE);
+ }
+ }
+ if (va == eva)
+ va = sva;
+ if (pmap_unuse_pt(pmap, sva, l2e, free)) {
+ sva += L3_SIZE;
+ break;
+ }
+ }
+ if (va != eva)
+ pmap_invalidate_range(pmap, va, sva);
+}
+
+/*
* Remove the given range of addresses from the specified map.
*
* It is assumed that the start and end are properly
@@ -2519,9 +2595,9 @@ void
pmap_remove(pmap_t pmap, vm_offset_t sva, vm_offset_t eva)
{
struct rwlock *lock;
- vm_offset_t va, va_next;
+ vm_offset_t va_next;
pd_entry_t *l0, *l1, *l2;
- pt_entry_t l3_paddr, *l3;
+ pt_entry_t l3_paddr;
struct spglist free;
/*
@@ -2594,28 +2670,8 @@ pmap_remove(pmap_t pmap, vm_offset_t sva, vm_offset_t
if (va_next > eva)
svn commit: r349183 - head/sys/arm64/arm64
Author: alc
Date: Wed Jun 19 03:33:00 2019
New Revision: 349183
URL: https://svnweb.freebsd.org/changeset/base/349183
Log:
Correct an error in r349122. pmap_unwire() should update the pmap's wired
count, not its resident count.
X-MFC with: r349122
Modified:
head/sys/arm64/arm64/pmap.c
Modified: head/sys/arm64/arm64/pmap.c
==
--- head/sys/arm64/arm64/pmap.c Wed Jun 19 01:28:13 2019(r349182)
+++ head/sys/arm64/arm64/pmap.c Wed Jun 19 03:33:00 2019(r349183)
@@ -3777,8 +3777,8 @@ pmap_unwire(pmap_t pmap, vm_offset_t sva, vm_offset_t
*/
if (sva + L2_SIZE == va_next && eva >= va_next) {
atomic_clear_64(l2, ATTR_SW_WIRED);
- pmap_resident_count_dec(pmap, L2_SIZE /
- PAGE_SIZE);
+ pmap->pm_stats.wired_count -= L2_SIZE /
+ PAGE_SIZE;
continue;
} else if (pmap_demote_l2(pmap, l2, sva) == NULL)
panic("pmap_unwire: demotion failed");
___
svn-src-all@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-all
To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"
svn commit: r349129 - head/sys/arm64/arm64
Author: alc Date: Mon Jun 17 01:58:25 2019 New Revision: 349129 URL: https://svnweb.freebsd.org/changeset/base/349129 Log: Eliminate a redundant call to pmap_invalidate_page() from pmap_ts_referenced(). MFC after:14 days Differential Revision:https://reviews.freebsd.org/D12725 Modified: head/sys/arm64/arm64/pmap.c Modified: head/sys/arm64/arm64/pmap.c == --- head/sys/arm64/arm64/pmap.c Mon Jun 17 00:58:49 2019(r349128) +++ head/sys/arm64/arm64/pmap.c Mon Jun 17 01:58:25 2019(r349129) @@ -4644,7 +4644,6 @@ small_mappings: */ pmap_remove_l3(pmap, pte, pv->pv_va, tpde, , ); - pmap_invalidate_page(pmap, pv->pv_va); cleared++; if (pvf == pv) pvf = NULL; ___ svn-src-all@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"
svn commit: r349122 - head/sys/arm64/arm64
Author: alc
Date: Sun Jun 16 22:13:27 2019
New Revision: 349122
URL: https://svnweb.freebsd.org/changeset/base/349122
Log:
Three changes to arm64's pmap_unwire():
Implement wiring changes on superpage mappings. Previously, a superpage
mapping was unconditionally demoted by pmap_unwire(), even if the wiring
change applied to the entire superpage mapping.
Rewrite a comment to use the arm64 names for bits in a page table entry.
Previously, the bits were referred to by their x86 names.
Use atomic_"op"_64() instead of atomic_"op"_long() to update a page table
entry in order to match the prevailing style in this file.
MFC after:10 days
Modified:
head/sys/arm64/arm64/pmap.c
Modified: head/sys/arm64/arm64/pmap.c
==
--- head/sys/arm64/arm64/pmap.c Sun Jun 16 21:56:45 2019(r349121)
+++ head/sys/arm64/arm64/pmap.c Sun Jun 16 22:13:27 2019(r349122)
@@ -3767,9 +3767,21 @@ pmap_unwire(pmap_t pmap, vm_offset_t sva, vm_offset_t
continue;
if ((pmap_load(l2) & ATTR_DESCR_MASK) == L2_BLOCK) {
- l3 = pmap_demote_l2(pmap, l2, sva);
- if (l3 == NULL)
+ if ((pmap_load(l2) & ATTR_SW_WIRED) == 0)
+ panic("pmap_unwire: l2 %#jx is missing "
+ "ATTR_SW_WIRED", (uintmax_t)pmap_load(l2));
+
+ /*
+* Are we unwiring the entire large page? If not,
+* demote the mapping and fall through.
+*/
+ if (sva + L2_SIZE == va_next && eva >= va_next) {
+ atomic_clear_64(l2, ATTR_SW_WIRED);
+ pmap_resident_count_dec(pmap, L2_SIZE /
+ PAGE_SIZE);
continue;
+ } else if (pmap_demote_l2(pmap, l2, sva) == NULL)
+ panic("pmap_unwire: demotion failed");
}
KASSERT((pmap_load(l2) & ATTR_DESCR_MASK) == L2_TABLE,
("pmap_unwire: Invalid l2 entry after demotion"));
@@ -3785,11 +3797,11 @@ pmap_unwire(pmap_t pmap, vm_offset_t sva, vm_offset_t
"ATTR_SW_WIRED", (uintmax_t)pmap_load(l3));
/*
-* PG_W must be cleared atomically. Although the pmap
-* lock synchronizes access to PG_W, another processor
-* could be setting PG_M and/or PG_A concurrently.
+* ATTR_SW_WIRED must be cleared atomically. Although
+* the pmap lock synchronizes access to ATTR_SW_WIRED,
+* the System MMU may write to the entry concurrently.
*/
- atomic_clear_long(l3, ATTR_SW_WIRED);
+ atomic_clear_64(l3, ATTR_SW_WIRED);
pmap->pm_stats.wired_count--;
}
}
___
svn-src-all@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-all
To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"
svn commit: r349117 - head/sys/arm64/arm64
Author: alc
Date: Sun Jun 16 16:45:01 2019
New Revision: 349117
URL: https://svnweb.freebsd.org/changeset/base/349117
Log:
Three enhancements to arm64's pmap_protect():
Implement protection changes on superpage mappings. Previously, a superpage
mapping was unconditionally demoted by pmap_protect(), even if the
protection change applied to the entire superpage mapping.
Precompute the bit mask describing the protection changes rather than
recomputing it for every page table entry that is changed.
Skip page table entries that already have the requested protection changes
in place.
Reviewed by: andrew, kib
MFC after:10 days
Differential Revision:https://reviews.freebsd.org/D20657
Modified:
head/sys/arm64/arm64/pmap.c
Modified: head/sys/arm64/arm64/pmap.c
==
--- head/sys/arm64/arm64/pmap.c Sun Jun 16 16:02:50 2019(r349116)
+++ head/sys/arm64/arm64/pmap.c Sun Jun 16 16:45:01 2019(r349117)
@@ -2729,6 +2729,51 @@ retry:
}
/*
+ * pmap_protect_l2: do the things to protect a 2MB page in a pmap
+ */
+static void
+pmap_protect_l2(pmap_t pmap, pt_entry_t *l2, vm_offset_t sva, pt_entry_t nbits)
+{
+ pd_entry_t old_l2;
+ vm_page_t m, mt;
+
+ PMAP_LOCK_ASSERT(pmap, MA_OWNED);
+ KASSERT((sva & L2_OFFSET) == 0,
+ ("pmap_protect_l2: sva is not 2mpage aligned"));
+ old_l2 = pmap_load(l2);
+ KASSERT((old_l2 & ATTR_DESCR_MASK) == L2_BLOCK,
+ ("pmap_protect_l2: L2e %lx is not a block mapping", old_l2));
+
+ /*
+* Return if the L2 entry already has the desired access restrictions
+* in place.
+*/
+ if ((old_l2 | nbits) == old_l2)
+ return;
+
+ /*
+* When a dirty read/write superpage mapping is write protected,
+* update the dirty field of each of the superpage's constituent 4KB
+* pages.
+*/
+ if ((nbits & ATTR_AP(ATTR_AP_RO)) != 0 &&
+ (old_l2 & ATTR_SW_MANAGED) != 0 &&
+ pmap_page_dirty(old_l2)) {
+ m = PHYS_TO_VM_PAGE(old_l2 & ~ATTR_MASK);
+ for (mt = m; mt < [L2_SIZE / PAGE_SIZE]; mt++)
+ vm_page_dirty(mt);
+ }
+
+ pmap_set(l2, nbits);
+
+ /*
+* Since a promotion must break the 4KB page mappings before making
+* the 2MB page mapping, a pmap_invalidate_page() suffices.
+*/
+ pmap_invalidate_page(pmap, sva);
+}
+
+/*
* Set the physical protection on the
* specified range of this map as requested.
*/
@@ -2745,8 +2790,12 @@ pmap_protect(pmap_t pmap, vm_offset_t sva, vm_offset_t
return;
}
- if ((prot & (VM_PROT_WRITE | VM_PROT_EXECUTE)) ==
- (VM_PROT_WRITE | VM_PROT_EXECUTE))
+ nbits = 0;
+ if ((prot & VM_PROT_WRITE) == 0)
+ nbits |= ATTR_AP(ATTR_AP_RO);
+ if ((prot & VM_PROT_EXECUTE) == 0)
+ nbits |= ATTR_XN;
+ if (nbits == 0)
return;
PMAP_LOCK(pmap);
@@ -2777,9 +2826,11 @@ pmap_protect(pmap_t pmap, vm_offset_t sva, vm_offset_t
continue;
if ((pmap_load(l2) & ATTR_DESCR_MASK) == L2_BLOCK) {
- l3p = pmap_demote_l2(pmap, l2, sva);
- if (l3p == NULL)
+ if (sva + L2_SIZE == va_next && eva >= va_next) {
+ pmap_protect_l2(pmap, l2, sva, nbits);
continue;
+ } else if (pmap_demote_l2(pmap, l2, sva) == NULL)
+ continue;
}
KASSERT((pmap_load(l2) & ATTR_DESCR_MASK) == L2_TABLE,
("pmap_protect: Invalid L2 entry after demotion"));
@@ -2790,8 +2841,16 @@ pmap_protect(pmap_t pmap, vm_offset_t sva, vm_offset_t
va = va_next;
for (l3p = pmap_l2_to_l3(l2, sva); sva != va_next; l3p++,
sva += L3_SIZE) {
+ /*
+* Go to the next L3 entry if the current one is
+* invalid or already has the desired access
+* restrictions in place. (The latter case occurs
+* frequently. For example, in a "buildworld"
+* workload, almost 1 out of 4 L3 entries already
+* have the desired restrictions.)
+*/
l3 = pmap_load(l3p);
- if (!pmap_l3_valid(l3)) {
+ if (!pmap_l3_valid(l3) || (l3 | nbits) == l3) {
if (va != va_next) {
pmap_invalidate_range(pmap, va, sva);
va = va_next;
@@ -2801,17 +2860,14 @@ pmap_protect(pmap_t pmap, vm_offset_t sva,
svn commit: r349070 - head/sys/arm64/arm64
Author: alc
Date: Sat Jun 15 17:26:42 2019
New Revision: 349070
URL: https://svnweb.freebsd.org/changeset/base/349070
Log:
Previously, when pmap_remove_pages() destroyed a dirty superpage mapping,
it only called vm_page_dirty() on the first of the superpage's constituent
4KB pages. This revision corrects that error, calling vm_page_dirty() on
all of superpage's constituent 4KB pages.
MFC after:3 days
Modified:
head/sys/arm64/arm64/pmap.c
Modified: head/sys/arm64/arm64/pmap.c
==
--- head/sys/arm64/arm64/pmap.c Sat Jun 15 17:08:39 2019(r349069)
+++ head/sys/arm64/arm64/pmap.c Sat Jun 15 17:26:42 2019(r349070)
@@ -4067,7 +4067,7 @@ pmap_remove_pages(pmap_t pmap)
switch (lvl) {
case 1:
for (mt = m; mt < [L2_SIZE /
PAGE_SIZE]; mt++)
- vm_page_dirty(m);
+ vm_page_dirty(mt);
break;
case 2:
vm_page_dirty(m);
___
svn-src-all@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-all
To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"
svn commit: r349042 - head/sys/arm64/arm64
Author: alc
Date: Fri Jun 14 22:06:43 2019
New Revision: 349042
URL: https://svnweb.freebsd.org/changeset/base/349042
Log:
Batch the TLB invalidations that are performed by pmap_protect() rather
than performing them one at a time.
MFC after:10 days
Modified:
head/sys/arm64/arm64/pmap.c
Modified: head/sys/arm64/arm64/pmap.c
==
--- head/sys/arm64/arm64/pmap.c Fri Jun 14 20:35:37 2019(r349041)
+++ head/sys/arm64/arm64/pmap.c Fri Jun 14 22:06:43 2019(r349042)
@@ -2791,8 +2791,15 @@ pmap_protect(pmap_t pmap, vm_offset_t sva, vm_offset_t
for (l3p = pmap_l2_to_l3(l2, sva); sva != va_next; l3p++,
sva += L3_SIZE) {
l3 = pmap_load(l3p);
- if (!pmap_l3_valid(l3))
+ if (!pmap_l3_valid(l3)) {
+ if (va != va_next) {
+ pmap_invalidate_range(pmap, va, sva);
+ va = va_next;
+ }
continue;
+ }
+ if (va == va_next)
+ va = sva;
nbits = 0;
if ((prot & VM_PROT_WRITE) == 0) {
@@ -2807,9 +2814,9 @@ pmap_protect(pmap_t pmap, vm_offset_t sva, vm_offset_t
nbits |= ATTR_XN;
pmap_set(l3p, nbits);
- /* XXX: Use pmap_invalidate_range */
- pmap_invalidate_page(pmap, sva);
}
+ if (va != va_next)
+ pmap_invalidate_range(pmap, va, sva);
}
PMAP_UNLOCK(pmap);
}
___
svn-src-all@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-all
To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"
svn commit: r349031 - head/sys/arm64/arm64
Author: alc
Date: Fri Jun 14 04:01:08 2019
New Revision: 349031
URL: https://svnweb.freebsd.org/changeset/base/349031
Log:
Change the arm64 pmap so that updates to the global count of wired pages are
not performed directly by the pmap. Instead, they are performed by
vm_page_free_pages_toq(). (This is the same approach that we use on x86.)
Reviewed by: kib, markj
MFC after:10 days
Differential Revision:https://reviews.freebsd.org/D20627
Modified:
head/sys/arm64/arm64/pmap.c
Modified: head/sys/arm64/arm64/pmap.c
==
--- head/sys/arm64/arm64/pmap.c Fri Jun 14 03:15:54 2019(r349030)
+++ head/sys/arm64/arm64/pmap.c Fri Jun 14 04:01:08 2019(r349031)
@@ -1410,8 +1410,6 @@ _pmap_unwire_l3(pmap_t pmap, vm_offset_t va, vm_page_t
}
pmap_invalidate_page(pmap, va);
- vm_wire_sub(1);
-
/*
* Put page on a list so that it is released after
* *ALL* TLB shootdown is done
@@ -2030,9 +2028,8 @@ next_chunk:
SLIST_REMOVE_HEAD(, plinks.s.ss);
/* Recycle a freed page table page. */
m_pc->wire_count = 1;
- vm_wire_add(1);
}
- vm_page_free_pages_toq(, false);
+ vm_page_free_pages_toq(, true);
return (m_pc);
}
@@ -2470,8 +2467,7 @@ pmap_remove_l2(pmap_t pmap, pt_entry_t *l2, vm_offset_
pmap_resident_count_dec(pmap, 1);
KASSERT(ml3->wire_count == NL3PG,
("pmap_remove_l2: l3 page wire count error"));
- ml3->wire_count = 1;
- vm_page_unwire_noq(ml3);
+ ml3->wire_count = 0;
pmap_add_delayed_free_list(ml3, free, FALSE);
}
}
@@ -2624,7 +2620,7 @@ pmap_remove(pmap_t pmap, vm_offset_t sva, vm_offset_t
if (lock != NULL)
rw_wunlock(lock);
PMAP_UNLOCK(pmap);
- vm_page_free_pages_toq(, false);
+ vm_page_free_pages_toq(, true);
}
/*
@@ -2729,7 +2725,7 @@ retry:
}
vm_page_aflag_clear(m, PGA_WRITEABLE);
rw_wunlock(lock);
- vm_page_free_pages_toq(, false);
+ vm_page_free_pages_toq(, true);
}
/*
@@ -3397,7 +3393,7 @@ pmap_enter_l2(pmap_t pmap, vm_offset_t va, pd_entry_t
lockp) != 0)
break;
}
- vm_page_free_pages_toq(, false);
+ vm_page_free_pages_toq(, true);
if (va >= VM_MAXUSER_ADDRESS) {
/*
* Both pmap_remove_l2() and pmap_remove_l3() will
@@ -3425,7 +3421,7 @@ pmap_enter_l2(pmap_t pmap, vm_offset_t va, pd_entry_t
* Invalidate those entries.
*/
pmap_invalidate_page(pmap, va);
- vm_page_free_pages_toq(, false);
+ vm_page_free_pages_toq(, true);
}
CTR2(KTR_PMAP,
"pmap_enter_l2: failure for va %#lx in pmap %p",
@@ -3614,7 +3610,7 @@ pmap_enter_quick_locked(pmap_t pmap, vm_offset_t va, v
SLIST_INIT();
if (pmap_unwire_l3(pmap, va, mpte, )) {
pmap_invalidate_page(pmap, va);
- vm_page_free_pages_toq(, false);
+ vm_page_free_pages_toq(, true);
}
mpte = NULL;
}
@@ -4097,8 +4093,7 @@ pmap_remove_pages(pmap_t pmap)
pmap_resident_count_dec(pmap,1);
KASSERT(ml3->wire_count ==
NL3PG,
("pmap_remove_pages: l3
page wire count error"));
- ml3->wire_count = 1;
- vm_page_unwire_noq(ml3);
+ ml3->wire_count = 0;
pmap_add_delayed_free_list(ml3,
, FALSE);
}
@@ -4136,7 +4131,7 @@ pmap_remove_pages(pmap_t pmap)
if (lock != NULL)
rw_wunlock(lock);
PMAP_UNLOCK(pmap);
- vm_page_free_pages_toq(, false);
+ vm_page_free_pages_toq(, true);
}
/*
@@ -4596,7 +4591,7 @@ small_mappings:
not_cleared < PMAP_TS_REFERENCED_MAX);
out:
rw_wunlock(lock);
- vm_page_free_pages_toq(, false);
+ vm_page_free_pages_toq(, true);
return (cleared + not_cleared);
}
@@ -5019,7 +5014,7 @@ pmap_demote_l2_abort(pmap_t pmap, vm_offset_t
svn commit: r349003 - head/sys/arm64/arm64
Author: alc
Date: Wed Jun 12 20:38:49 2019
New Revision: 349003
URL: https://svnweb.freebsd.org/changeset/base/349003
Log:
Change pmap_demote_l2_locked() so that it removes the superpage mapping on a
demotion failure. Otherwise, some callers to pmap_demote_l2_locked(), such
as pmap_protect(), may leave an incorrect mapping in place on a demotion
failure.
Change pmap_demote_l2_locked() so that it handles addresses that are not
superpage aligned. Some callers to pmap_demote_l2_locked(), such as
pmap_protect(), may not pass a superpage aligned address.
Change pmap_enter_l2() so that it correctly calls vm_page_free_pages_toq().
The arm64 pmap is updating the count of wired pages when freeing page table
pages, so pmap_enter_l2() should pass false to vm_page_free_pages_toq().
Optimize TLB invalidation in pmap_remove_l2().
Reviewed by: kib, markj (an earlier version)
Discussed with: andrew
MFC after:3 weeks
Differential Revision:https://reviews.freebsd.org/D20585
Modified:
head/sys/arm64/arm64/pmap.c
Modified: head/sys/arm64/arm64/pmap.c
==
--- head/sys/arm64/arm64/pmap.c Wed Jun 12 19:31:26 2019(r349002)
+++ head/sys/arm64/arm64/pmap.c Wed Jun 12 20:38:49 2019(r349003)
@@ -2267,6 +2267,8 @@ pmap_pv_demote_l2(pmap_t pmap, vm_offset_t va, vm_padd
int bit, field;
PMAP_LOCK_ASSERT(pmap, MA_OWNED);
+ KASSERT((va & L2_OFFSET) == 0,
+ ("pmap_pv_demote_l2: va is not 2mpage aligned"));
KASSERT((pa & L2_OFFSET) == 0,
("pmap_pv_demote_l2: pa is not 2mpage aligned"));
CHANGE_PV_LIST_LOCK_TO_PHYS(lockp, pa);
@@ -2277,7 +2279,6 @@ pmap_pv_demote_l2(pmap_t pmap, vm_offset_t va, vm_padd
* must not be released until the last pv entry is reinstantiated.
*/
pvh = pa_to_pvh(pa);
- va = va & ~L2_OFFSET;
pv = pmap_pvh_remove(pvh, pmap, va);
KASSERT(pv != NULL, ("pmap_pv_demote_l2: pv not found"));
m = PHYS_TO_VM_PAGE(pa);
@@ -2433,7 +2434,13 @@ pmap_remove_l2(pmap_t pmap, pt_entry_t *l2, vm_offset_
old_l2 = pmap_load_clear(l2);
KASSERT((old_l2 & ATTR_DESCR_MASK) == L2_BLOCK,
("pmap_remove_l2: L2e %lx is not a block mapping", old_l2));
- pmap_invalidate_range(pmap, sva, sva + L2_SIZE);
+
+ /*
+* Since a promotion must break the 4KB page mappings before making
+* the 2MB page mapping, a pmap_invalidate_page() suffices.
+*/
+ pmap_invalidate_page(pmap, sva);
+
if (old_l2 & ATTR_SW_WIRED)
pmap->pm_stats.wired_count -= L2_SIZE / PAGE_SIZE;
pmap_resident_count_dec(pmap, L2_SIZE / PAGE_SIZE);
@@ -2571,8 +2578,8 @@ pmap_remove(pmap_t pmap, vm_offset_t sva, vm_offset_t
pmap_remove_l2(pmap, l2, sva, pmap_load(l1),
, );
continue;
- } else if (pmap_demote_l2_locked(pmap, l2,
- sva &~L2_OFFSET, ) == NULL)
+ } else if (pmap_demote_l2_locked(pmap, l2, sva,
+ ) == NULL)
continue;
l3_paddr = pmap_load(l2);
}
@@ -3052,8 +3059,7 @@ pmap_enter(pmap_t pmap, vm_offset_t va, vm_page_t m, v
if (pde != NULL && lvl == 1) {
l2 = pmap_l1_to_l2(pde, va);
if ((pmap_load(l2) & ATTR_DESCR_MASK) == L2_BLOCK &&
- (l3 = pmap_demote_l2_locked(pmap, l2, va & ~L2_OFFSET,
- )) != NULL) {
+ (l3 = pmap_demote_l2_locked(pmap, l2, va, )) != NULL) {
l3 = [pmap_l3_index(va)];
if (va < VM_MAXUSER_ADDRESS) {
mpte = PHYS_TO_VM_PAGE(
@@ -3391,7 +3397,7 @@ pmap_enter_l2(pmap_t pmap, vm_offset_t va, pd_entry_t
lockp) != 0)
break;
}
- vm_page_free_pages_toq(, true);
+ vm_page_free_pages_toq(, false);
if (va >= VM_MAXUSER_ADDRESS) {
/*
* Both pmap_remove_l2() and pmap_remove_l3() will
@@ -3419,7 +3425,7 @@ pmap_enter_l2(pmap_t pmap, vm_offset_t va, pd_entry_t
* Invalidate those entries.
*/
pmap_invalidate_page(pmap, va);
- vm_page_free_pages_toq(, true);
+ vm_page_free_pages_toq(, false);
}
CTR2(KTR_PMAP,
"pmap_enter_l2: failure for va %#lx in pmap %p",
@@ -4331,8 +4337,7 @@ retry_pv_loop:
va = pv->pv_va;
pte = pmap_pte(pmap,
svn commit: r348829 - head/sys/riscv/riscv
Author: alc
Date: Sun Jun 9 05:55:58 2019
New Revision: 348829
URL: https://svnweb.freebsd.org/changeset/base/348829
Log:
Correct a new KASSERT() in r348828.
X-MFC with: r348828
Modified:
head/sys/riscv/riscv/pmap.c
Modified: head/sys/riscv/riscv/pmap.c
==
--- head/sys/riscv/riscv/pmap.c Sun Jun 9 03:36:10 2019(r348828)
+++ head/sys/riscv/riscv/pmap.c Sun Jun 9 05:55:58 2019(r348829)
@@ -3564,8 +3564,8 @@ pmap_remove_pages_pv(pmap_t pmap, vm_page_t m, pv_entr
}
mpte = pmap_remove_pt_page(pmap, pv->pv_va);
if (mpte != NULL) {
- KASSERT(ml3->valid == VM_PAGE_BITS_ALL,
- ("pmap_remove_pages: l3 page not promoted"));
+ KASSERT(mpte->valid == VM_PAGE_BITS_ALL,
+ ("pmap_remove_pages: pte page not promoted"));
pmap_resident_count_dec(pmap, 1);
KASSERT(mpte->wire_count == Ln_ENTRIES,
("pmap_remove_pages: pte page wire count error"));
___
svn-src-all@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-all
To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"
svn commit: r348828 - in head/sys: amd64/amd64 arm64/arm64 i386/i386 riscv/riscv
Author: alc
Date: Sun Jun 9 03:36:10 2019
New Revision: 348828
URL: https://svnweb.freebsd.org/changeset/base/348828
Log:
Implement an alternative solution to the amd64 and i386 pmap problem that we
previously addressed in r348246.
This pmap problem also exists on arm64 and riscv. However, the original
solution developed for amd64 and i386 cannot be used on arm64 and riscv. In
particular, arm64 and riscv do not define a PG_PROMOTED flag in their level
2 PTEs. (A PG_PROMOTED flag makes no sense on arm64, where unlike x86 or
riscv we are required to break the old 4KB mappings before making the 2MB
mapping; and on riscv there are no unused bits in the PTE to define a
PG_PROMOTED flag.)
This commit implements an alternative solution that can be used on all four
architectures. Moreover, this solution has two other advantages. First, on
older AMD processors that required the Erratum 383 workaround, it is less
costly. Specifically, it avoids unnecessary calls to pmap_fill_ptp() on a
superpage demotion. Second, it enables the elimination of some calls to
pagezero() in pmap_kernel_remove_{l2,pde}().
In addition, remove a related stale comment from pmap_enter_{l2,pde}().
Reviewed by: kib, markj (an earlier version)
MFC after:1 week
Differential Revision:https://reviews.freebsd.org/D20538
Modified:
head/sys/amd64/amd64/pmap.c
head/sys/arm64/arm64/pmap.c
head/sys/i386/i386/pmap.c
head/sys/riscv/riscv/pmap.c
Modified: head/sys/amd64/amd64/pmap.c
==
--- head/sys/amd64/amd64/pmap.c Sun Jun 9 03:31:07 2019(r348827)
+++ head/sys/amd64/amd64/pmap.c Sun Jun 9 03:36:10 2019(r348828)
@@ -1053,7 +1053,7 @@ static intpmap_enter_pde(pmap_t pmap, vm_offset_t
va,
static vm_page_t pmap_enter_quick_locked(pmap_t pmap, vm_offset_t va,
vm_page_t m, vm_prot_t prot, vm_page_t mpte, struct rwlock **lockp);
static void pmap_fill_ptp(pt_entry_t *firstpte, pt_entry_t newpte);
-static int pmap_insert_pt_page(pmap_t pmap, vm_page_t mpte);
+static int pmap_insert_pt_page(pmap_t pmap, vm_page_t mpte, bool promoted);
static void pmap_invalidate_cache_range_selfsnoop(vm_offset_t sva,
vm_offset_t eva);
static void pmap_invalidate_cache_range_all(vm_offset_t sva,
@@ -1757,8 +1757,13 @@ pmap_init(void)
mpte->pindex = pmap_pde_pindex(KERNBASE) + i;
mpte->phys_addr = KPTphys + (i << PAGE_SHIFT);
mpte->wire_count = 1;
+
+ /*
+* Collect the page table pages that were replaced by a 2MB
+* page in create_pagetables(). They are zero filled.
+*/
if (i << PDRSHIFT < KERNend &&
- pmap_insert_pt_page(kernel_pmap, mpte))
+ pmap_insert_pt_page(kernel_pmap, mpte, false))
panic("pmap_init: pmap_insert_pt_page failed");
}
PMAP_UNLOCK(kernel_pmap);
@@ -3129,12 +3134,15 @@ pmap_add_delayed_free_list(vm_page_t m, struct spglist
* of idle page table pages. Each of a pmap's page table pages is responsible
* for mapping a distinct range of virtual addresses. The pmap's collection is
* ordered by this virtual address range.
+ *
+ * If "promoted" is false, then the page table page "mpte" must be zero filled.
*/
static __inline int
-pmap_insert_pt_page(pmap_t pmap, vm_page_t mpte)
+pmap_insert_pt_page(pmap_t pmap, vm_page_t mpte, bool promoted)
{
PMAP_LOCK_ASSERT(pmap, MA_OWNED);
+ mpte->valid = promoted ? VM_PAGE_BITS_ALL : 0;
return (vm_radix_insert(>pm_root, mpte));
}
@@ -4626,7 +4634,7 @@ pmap_demote_pde_locked(pmap_t pmap, pd_entry_t *pde, v
* If the page table page is not leftover from an earlier promotion,
* initialize it.
*/
- if ((oldpde & PG_PROMOTED) == 0)
+ if (mpte->valid == 0)
pmap_fill_ptp(firstpte, newpte);
pmap_demote_pde_check(firstpte, newpte);
@@ -4699,9 +4707,11 @@ pmap_remove_kernel_pde(pmap_t pmap, pd_entry_t *pde, v
newpde = mptepa | X86_PG_M | X86_PG_A | X86_PG_RW | X86_PG_V;
/*
-* Initialize the page table page.
+* If this page table page was unmapped by a promotion, then it
+* contains valid mappings. Zero it to invalidate those mappings.
*/
- pagezero((void *)PHYS_TO_DMAP(mptepa));
+ if (mpte->valid != 0)
+ pagezero((void *)PHYS_TO_DMAP(mptepa));
/*
* Demote the mapping.
@@ -4766,6 +4776,8 @@ pmap_remove_pde(pmap_t pmap, pd_entry_t *pdq, vm_offse
} else {
mpte = pmap_remove_pt_page(pmap, sva);
if (mpte != NULL) {
+ KASSERT(mpte->valid == VM_PAGE_BITS_ALL,
+ ("pmap_remove_pde: pte page not promoted"));
pmap_resident_count_dec(pmap, 1);
svn commit: r348630 - head/sys/amd64/amd64
Author: alc
Date: Tue Jun 4 16:21:14 2019
New Revision: 348630
URL: https://svnweb.freebsd.org/changeset/base/348630
Log:
The changes to pmap_demote_pde_locked()'s control flow in r348476 resulted
in the loss of a KASSERT that guarded against the invalidation a wired
mapping. Restore this KASSERT.
Remove an unnecessary KASSERT from pmap_demote_pde_locked(). It guards
against a state that was already handled at the start of the function.
Reviewed by: kib
X-MFC with: r348476
Modified:
head/sys/amd64/amd64/pmap.c
Modified: head/sys/amd64/amd64/pmap.c
==
--- head/sys/amd64/amd64/pmap.c Tue Jun 4 16:07:01 2019(r348629)
+++ head/sys/amd64/amd64/pmap.c Tue Jun 4 16:21:14 2019(r348630)
@@ -4565,6 +4565,8 @@ pmap_demote_pde_locked(pmap_t pmap, pd_entry_t *pde, v
* mapping was never accessed.
*/
if ((oldpde & PG_A) == 0) {
+ KASSERT((oldpde & PG_W) == 0,
+ ("pmap_demote_pde: a wired mapping is missing PG_A"));
pmap_demote_pde_abort(pmap, va, pde, oldpde, lockp);
return (FALSE);
}
@@ -4615,8 +4617,6 @@ pmap_demote_pde_locked(pmap_t pmap, pd_entry_t *pde, v
mptepa = VM_PAGE_TO_PHYS(mpte);
firstpte = (pt_entry_t *)PHYS_TO_DMAP(mptepa);
newpde = mptepa | PG_M | PG_A | (oldpde & PG_U) | PG_RW | PG_V;
- KASSERT((oldpde & PG_A) != 0,
- ("pmap_demote_pde: oldpde is missing PG_A"));
KASSERT((oldpde & (PG_M | PG_RW)) != PG_RW,
("pmap_demote_pde: oldpde is missing PG_M"));
newpte = oldpde & ~PG_PS;
___
svn-src-all@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-all
To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"
svn commit: r348525 - head/sys/vm
Author: alc
Date: Mon Jun 3 05:15:36 2019
New Revision: 348525
URL: https://svnweb.freebsd.org/changeset/base/348525
Log:
Retire vm_reserv_extend_{contig,page}(). These functions were introduced
as part of a false start toward fine-grained reservation locking. In the
end, they were not needed, so eliminate them.
Order the parameters to vm_reserv_alloc_{contig,page}() consistently with
the vm_page functions that call them.
Update the comments about the locking requirements for
vm_reserv_alloc_{contig,page}(). They no longer require a free page
queues lock.
Wrap several lines that became too long after the "req" and "domain"
parameters were added to vm_reserv_alloc_{contig,page}().
Reviewed by: kib, markj
MFC after:1 week
Differential Revision:https://reviews.freebsd.org/D20492
Modified:
head/sys/vm/vm_page.c
head/sys/vm/vm_reserv.c
head/sys/vm/vm_reserv.h
Modified: head/sys/vm/vm_page.c
==
--- head/sys/vm/vm_page.c Mon Jun 3 04:38:07 2019(r348524)
+++ head/sys/vm/vm_page.c Mon Jun 3 05:15:36 2019(r348525)
@@ -1834,8 +1834,8 @@ again:
* Can we allocate the page from a reservation?
*/
if (vm_object_reserv(object) &&
- ((m = vm_reserv_extend(req, object, pindex, domain, mpred)) != NULL
||
- (m = vm_reserv_alloc_page(req, object, pindex, domain, mpred)) !=
NULL)) {
+ (m = vm_reserv_alloc_page(object, pindex, domain, req, mpred)) !=
+ NULL) {
domain = vm_phys_domain(m);
vmd = VM_DOMAIN(domain);
goto found;
@@ -2039,10 +2039,8 @@ again:
* Can we allocate the pages from a reservation?
*/
if (vm_object_reserv(object) &&
- ((m_ret = vm_reserv_extend_contig(req, object, pindex, domain,
- npages, low, high, alignment, boundary, mpred)) != NULL ||
- (m_ret = vm_reserv_alloc_contig(req, object, pindex, domain,
- npages, low, high, alignment, boundary, mpred)) != NULL)) {
+ (m_ret = vm_reserv_alloc_contig(object, pindex, domain, req,
+ mpred, npages, low, high, alignment, boundary)) != NULL) {
domain = vm_phys_domain(m_ret);
vmd = VM_DOMAIN(domain);
goto found;
Modified: head/sys/vm/vm_reserv.c
==
--- head/sys/vm/vm_reserv.c Mon Jun 3 04:38:07 2019(r348524)
+++ head/sys/vm/vm_reserv.c Mon Jun 3 05:15:36 2019(r348525)
@@ -576,95 +576,8 @@ vm_reserv_populate(vm_reserv_t rv, int index)
}
/*
- * Attempts to allocate a contiguous set of physical pages from existing
- * reservations. See vm_reserv_alloc_contig() for a description of the
- * function's parameters.
- *
- * The page "mpred" must immediately precede the offset "pindex" within the
- * specified object.
- *
- * The object must be locked.
- */
-vm_page_t
-vm_reserv_extend_contig(int req, vm_object_t object, vm_pindex_t pindex,
-int domain, u_long npages, vm_paddr_t low, vm_paddr_t high,
-u_long alignment, vm_paddr_t boundary, vm_page_t mpred)
-{
- struct vm_domain *vmd;
- vm_paddr_t pa, size;
- vm_page_t m, msucc;
- vm_reserv_t rv;
- int i, index;
-
- VM_OBJECT_ASSERT_WLOCKED(object);
- KASSERT(npages != 0, ("vm_reserv_alloc_contig: npages is 0"));
-
- /*
-* Is a reservation fundamentally impossible?
-*/
- if (pindex < VM_RESERV_INDEX(object, pindex) ||
- pindex + npages > object->size || object->resident_page_count == 0)
- return (NULL);
-
- /*
-* All reservations of a particular size have the same alignment.
-* Assuming that the first page is allocated from a reservation, the
-* least significant bits of its physical address can be determined
-* from its offset from the beginning of the reservation and the size
-* of the reservation.
-*
-* Could the specified index within a reservation of the smallest
-* possible size satisfy the alignment and boundary requirements?
-*/
- pa = VM_RESERV_INDEX(object, pindex) << PAGE_SHIFT;
- if ((pa & (alignment - 1)) != 0)
- return (NULL);
- size = npages << PAGE_SHIFT;
- if (((pa ^ (pa + size - 1)) & ~(boundary - 1)) != 0)
- return (NULL);
-
- /*
-* Look for an existing reservation.
-*/
- rv = vm_reserv_from_object(object, pindex, mpred, );
- if (rv == NULL)
- return (NULL);
- KASSERT(object != kernel_object || rv->domain == domain,
- ("vm_reserv_extend_contig: Domain mismatch from reservation."));
- index = VM_RESERV_INDEX(object, pindex);
- /* Does the allocation fit within the reservation? */
- if
svn commit: r341766 - head/sys/kern
Author: alc
Date: Sun Dec 9 17:55:10 2018
New Revision: 341766
URL: https://svnweb.freebsd.org/changeset/base/341766
Log:
blst_leaf_alloc updates bighint for a leaf when an allocation is successful
and includes the last block represented by the leaf. The reasoning is that,
if the last block is included, then there must be no solution before that
one in the leaf, so the leaf cannot provide an allocation that big again;
indeed, the leaf cannot provide a solution bigger than range1.
Which is all correct, except that if the value of blk passed in did not
represent the first block of the leaf, because the cursor was pointing to
the middle of the leaf, then a possible solution before the cursor may have
been ignored, and bighint cannot be updated.
Consider the sequence allocate 63 (returning address 0), free 0,63 (freeing
that same block, and allocate 1 (returning 63). The result is that one
block is allocated from the first leaf, and the value of bighint is 0, so
that nothing can be allocated from that leaf until the only block allocated
from that leaf is freed. This change detects that skipped-over solution,
and when there is one it makes sure that the value of bighint is not changed
when the last block is allocated.
Submitted by: Doug Moore
Tested by:pho
X-MFC with: r340402
Differential Revision:https://reviews.freebsd.org/D18474
Modified:
head/sys/kern/subr_blist.c
Modified: head/sys/kern/subr_blist.c
==
--- head/sys/kern/subr_blist.c Sun Dec 9 15:34:20 2018(r341765)
+++ head/sys/kern/subr_blist.c Sun Dec 9 17:55:10 2018(r341766)
@@ -644,14 +644,14 @@ blst_next_leaf_alloc(blmeta_t *scan, daddr_t blk, int
/*
* BLST_LEAF_ALLOC() - allocate at a leaf in the radix tree (a bitmap).
*
- * This is the core of the allocator and is optimized for the
- * BLIST_BMAP_RADIX block allocation case. Otherwise, execution
- * time is proportional to log2(count) + bitpos time.
+ * This function is the core of the allocator. Its execution time is
+ * proportional to log(count), plus height of the tree if the allocation
+ * crosses a leaf boundary.
*/
static daddr_t
blst_leaf_alloc(blmeta_t *scan, daddr_t blk, int count)
{
- u_daddr_t mask;
+ u_daddr_t cursor_mask, mask;
int count1, hi, lo, num_shifts, range1, range_ext;
range1 = 0;
@@ -661,14 +661,14 @@ blst_leaf_alloc(blmeta_t *scan, daddr_t blk, int count
while ((-mask & ~mask) != 0 && num_shifts > 0) {
/*
* If bit i is set in mask, then bits in [i, i+range1] are set
-* in scan->bm_bitmap. The value of range1 is equal to
-* count1 >> num_shifts. Grow range and reduce num_shifts to 0,
-* while preserving these invariants. The updates to mask leave
-* fewer bits set, but each bit that remains set represents a
-* longer string of consecutive bits set in scan->bm_bitmap.
-* If more updates to mask cannot clear more bits, because mask
-* is partitioned with all 0 bits preceding all 1 bits, the loop
-* terminates immediately.
+* in scan->bm_bitmap. The value of range1 is equal to count1
+* >> num_shifts. Grow range1 and reduce num_shifts to 0,
+* while preserving these invariants. The updates to mask
+* leave fewer bits set, but each bit that remains set
+* represents a longer string of consecutive bits set in
+* scan->bm_bitmap. If more updates to mask cannot clear more
+* bits, because mask is partitioned with all 0 bits preceding
+* all 1 bits, the loop terminates immediately.
*/
num_shifts--;
range_ext = range1 + ((count1 >> num_shifts) & 1);
@@ -691,10 +691,23 @@ blst_leaf_alloc(blmeta_t *scan, daddr_t blk, int count
}
/* Discard any candidates that appear before blk. */
- mask &= (u_daddr_t)-1 << (blk & BLIST_BMAP_MASK);
- if (mask == 0)
- return (SWAPBLK_NONE);
+ if ((blk & BLIST_BMAP_MASK) != 0) {
+ cursor_mask = mask & bitrange(0, blk & BLIST_BMAP_MASK);
+ if (cursor_mask != 0) {
+ mask ^= cursor_mask;
+ if (mask == 0)
+ return (SWAPBLK_NONE);
+ /*
+* Bighint change for last block allocation cannot
+* assume that any other blocks are allocated, so the
+* bighint cannot be reduced much.
+*/
+ range1 = BLIST_MAX_ALLOC - 1;
+ }
+ blk &= ~BLIST_BMAP_MASK;
+ }
+
/*
svn commit: r341602 - head/sys/kern
Author: alc
Date: Wed Dec 5 18:26:40 2018
New Revision: 341602
URL: https://svnweb.freebsd.org/changeset/base/341602
Log:
Terminate a blist_alloc search when a blst_meta_alloc call fails with
cursor == 0.
Every call to blst_meta_alloc but the one at the root is made only when the
meta-node is known to include a free block, so that either the allocation
will succeed, the node hint will be updated, or the last block of the meta-
node range is, and remains, free. But the call at the root is made without
checking that there is a free block, so in the case that every block is
allocated, there is no hint update to prevent the current code from looping
forever.
Submitted by: Doug Moore
Reported by: pho
Reviewed by: pho
Tested by:pho
X-MFC with: r340402
Differential Revision:https://reviews.freebsd.org/D17999
Modified:
head/sys/kern/subr_blist.c
Modified: head/sys/kern/subr_blist.c
==
--- head/sys/kern/subr_blist.c Wed Dec 5 18:19:29 2018(r341601)
+++ head/sys/kern/subr_blist.c Wed Dec 5 18:26:40 2018(r341602)
@@ -295,9 +295,9 @@ blist_alloc(blist_t bl, daddr_t count)
* This loop iterates at most twice. An allocation failure in the
* first iteration leads to a second iteration only if the cursor was
* non-zero. When the cursor is zero, an allocation failure will
-* reduce the hint, stopping further iterations.
+* stop further iterations.
*/
- while (count <= bl->bl_root->bm_bighint) {
+ for (;;) {
blk = blst_meta_alloc(bl->bl_root, bl->bl_cursor, count,
bl->bl_radix);
if (blk != SWAPBLK_NONE) {
@@ -306,10 +306,10 @@ blist_alloc(blist_t bl, daddr_t count)
if (bl->bl_cursor == bl->bl_blocks)
bl->bl_cursor = 0;
return (blk);
- }
+ } else if (bl->bl_cursor == 0)
+ return (SWAPBLK_NONE);
bl->bl_cursor = 0;
}
- return (SWAPBLK_NONE);
}
/*
___
svn-src-all@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-all
To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"
svn commit: r340914 - head/sys/kern
Author: alc Date: Sat Nov 24 21:52:10 2018 New Revision: 340914 URL: https://svnweb.freebsd.org/changeset/base/340914 Log: blist_meta_alloc assumes that mask=scan->bm_bitmap is nonzero. But if the cursor lies in the middle of the space that the meta node represents, then blanking the low bits of mask may make it zero, and break later code that expects a nonzero value. Add a test that returns failure if the mask has been cleared. Submitted by: Doug Moore Reported by: pho Tested by:pho X-MFC with: r340402 Differential Revision:https://reviews.freebsd.org/D18058 Modified: head/sys/kern/subr_blist.c Modified: head/sys/kern/subr_blist.c == --- head/sys/kern/subr_blist.c Sat Nov 24 21:23:12 2018(r340913) +++ head/sys/kern/subr_blist.c Sat Nov 24 21:52:10 2018(r340914) @@ -764,6 +764,8 @@ blst_meta_alloc(blmeta_t *scan, daddr_t cursor, daddr_ /* Discard any candidates that appear before cursor. */ digit = (cursor / radix) & BLIST_META_MASK; mask &= (u_daddr_t)-1 << digit; + if (mask == 0) + return (SWAPBLK_NONE); /* * If the first try is for a block that includes the cursor, pre-undo ___ svn-src-all@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"
svn commit: r340637 - head/sys/vm
Author: alc
Date: Mon Nov 19 17:17:23 2018
New Revision: 340637
URL: https://svnweb.freebsd.org/changeset/base/340637
Log:
Use swp_pager_isondev() throughout. Submitted by: o...@j.email.ne.jp
Change swp_pager_isondev()'s return type to bool.
Reviewed by: kib
MFC after:1 week
Differential Revision:https://reviews.freebsd.org/D16712
Modified:
head/sys/vm/swap_pager.c
Modified: head/sys/vm/swap_pager.c
==
--- head/sys/vm/swap_pager.cMon Nov 19 17:09:57 2018(r340636)
+++ head/sys/vm/swap_pager.cMon Nov 19 17:17:23 2018(r340637)
@@ -756,7 +756,7 @@ done:
return (blk);
}
-static int
+static bool
swp_pager_isondev(daddr_t blk, struct swdevt *sp)
{
@@ -770,7 +770,7 @@ swp_pager_strategy(struct buf *bp)
mtx_lock(_dev_mtx);
TAILQ_FOREACH(sp, , sw_list) {
- if (bp->b_blkno >= sp->sw_first && bp->b_blkno < sp->sw_end) {
+ if (swp_pager_isondev(bp->b_blkno, sp)) {
mtx_unlock(_dev_mtx);
if ((sp->sw_flags & SW_UNMAPPED) != 0 &&
unmapped_buf_allowed) {
@@ -804,7 +804,7 @@ swp_pager_freeswapspace(daddr_t blk, daddr_t npages)
return;
mtx_lock(_dev_mtx);
TAILQ_FOREACH(sp, , sw_list) {
- if (blk >= sp->sw_first && blk < sp->sw_end) {
+ if (swp_pager_isondev(blk, sp)) {
sp->sw_used -= npages;
/*
* If we are attempting to stop swapping on
___
svn-src-all@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-all
To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"
svn commit: r340546 - head/sys/vm
Author: alc
Date: Sun Nov 18 01:27:17 2018
New Revision: 340546
URL: https://svnweb.freebsd.org/changeset/base/340546
Log:
Tidy up vm_map_simplify_entry() and its recently introduced helper
functions. Notably, reflow the text of some comments so that they
occupy fewer lines, and introduce an assertion in one of the new
helper functions so that it is not misused by a future caller.
In collaboration with:Doug Moore
MFC after:1 week
Differential Revision:https://reviews.freebsd.org/D17635
Modified:
head/sys/vm/vm_map.c
Modified: head/sys/vm/vm_map.c
==
--- head/sys/vm/vm_map.cSun Nov 18 01:07:36 2018(r340545)
+++ head/sys/vm/vm_map.cSun Nov 18 01:27:17 2018(r340546)
@@ -1644,16 +1644,25 @@ vm_map_find_min(vm_map_t map, vm_object_t object, vm_o
}
}
+/*
+ * A map entry with any of the following flags set must not be merged with
+ * another entry.
+ */
+#defineMAP_ENTRY_NOMERGE_MASK (MAP_ENTRY_GROWS_DOWN |
MAP_ENTRY_GROWS_UP | \
+ MAP_ENTRY_IN_TRANSITION | MAP_ENTRY_IS_SUB_MAP)
+
static bool
vm_map_mergeable_neighbors(vm_map_entry_t prev, vm_map_entry_t entry)
{
- vm_size_t prevsize;
- prevsize = prev->end - prev->start;
+ KASSERT((prev->eflags & MAP_ENTRY_NOMERGE_MASK) == 0 ||
+ (entry->eflags & MAP_ENTRY_NOMERGE_MASK) == 0,
+ ("vm_map_mergeable_neighbors: neither %p nor %p are mergeable",
+ prev, entry));
return (prev->end == entry->start &&
prev->object.vm_object == entry->object.vm_object &&
(prev->object.vm_object == NULL ||
- prev->offset + prevsize == entry->offset) &&
+ prev->offset + (prev->end - prev->start) == entry->offset) &&
prev->eflags == entry->eflags &&
prev->protection == entry->protection &&
prev->max_protection == entry->max_protection &&
@@ -1667,18 +1676,14 @@ vm_map_merged_neighbor_dispose(vm_map_t map, vm_map_en
{
/*
-* If the backing object is a vnode object,
-* vm_object_deallocate() calls vrele().
-* However, vrele() does not lock the vnode
-* because the vnode has additional
-* references. Thus, the map lock can be kept
-* without causing a lock-order reversal with
-* the vnode lock.
+* If the backing object is a vnode object, vm_object_deallocate()
+* calls vrele(). However, vrele() does not lock the vnode because
+* the vnode has additional references. Thus, the map lock can be
+* kept without causing a lock-order reversal with the vnode lock.
*
-* Since we count the number of virtual page
-* mappings in object->un_pager.vnp.writemappings,
-* the writemappings value should not be adjusted
-* when the entry is disposed of.
+* Since we count the number of virtual page mappings in
+* object->un_pager.vnp.writemappings, the writemappings value
+* should not be adjusted when the entry is disposed of.
*/
if (entry->object.vm_object != NULL)
vm_object_deallocate(entry->object.vm_object);
@@ -1704,10 +1709,8 @@ vm_map_simplify_entry(vm_map_t map, vm_map_entry_t ent
{
vm_map_entry_t next, prev;
- if ((entry->eflags & (MAP_ENTRY_GROWS_DOWN | MAP_ENTRY_GROWS_UP |
- MAP_ENTRY_IN_TRANSITION | MAP_ENTRY_IS_SUB_MAP)) != 0)
+ if ((entry->eflags & MAP_ENTRY_NOMERGE_MASK) != 0)
return;
-
prev = entry->prev;
if (vm_map_mergeable_neighbors(prev, entry)) {
vm_map_entry_unlink(map, prev);
@@ -1717,7 +1720,6 @@ vm_map_simplify_entry(vm_map_t map, vm_map_entry_t ent
vm_map_entry_resize_free(map, entry->prev);
vm_map_merged_neighbor_dispose(map, prev);
}
-
next = entry->next;
if (vm_map_mergeable_neighbors(entry, next)) {
vm_map_entry_unlink(map, next);
@@ -1726,6 +1728,7 @@ vm_map_simplify_entry(vm_map_t map, vm_map_entry_t ent
vm_map_merged_neighbor_dispose(map, next);
}
}
+
/*
* vm_map_clip_start: [ internal use only ]
*
___
svn-src-all@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-all
To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"
svn commit: r339819 - head/sys/vm
Author: alc Date: Sat Oct 27 17:49:46 2018 New Revision: 339819 URL: https://svnweb.freebsd.org/changeset/base/339819 Log: Eliminate typically pointless calls to vm_fault_prefault() on soft, copy- on-write faults. On a page fault, when we call vm_fault_prefault(), it probes the pmap and the shadow chain of vm objects to see if there are opportunities to create read and/or execute-only mappings to neighoring pages. For example, in the case of hard faults, such effort typically pays off, that is, mappings are created that eliminate future soft page faults. However, in the the case of soft, copy-on-write faults, the effort very rarely pays off. (See the review for some specific data.) Reviewed by: kib, markj MFC after:3 weeks Differential Revision:https://reviews.freebsd.org/D17367 Modified: head/sys/vm/vm_fault.c Modified: head/sys/vm/vm_fault.c == --- head/sys/vm/vm_fault.c Sat Oct 27 17:21:13 2018(r339818) +++ head/sys/vm/vm_fault.c Sat Oct 27 17:49:46 2018(r339819) @@ -1181,6 +1181,16 @@ readrest: */ vm_object_pip_wakeup(fs.object); VM_OBJECT_WUNLOCK(fs.object); + + /* +* We only try to prefault read-only mappings to the +* neighboring pages when this copy-on-write fault is +* a hard fault. In other cases, trying to prefault +* is typically wasted effort. +*/ + if (faultcount == 0) + faultcount = 1; + /* * Only use the new page below... */ ___ svn-src-all@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"
svn commit: r338912 - head/sys/vm
Author: alc
Date: Mon Sep 24 16:49:02 2018
New Revision: 338912
URL: https://svnweb.freebsd.org/changeset/base/338912
Log:
Passing UMA_ZONE_NOFREE to uma_zcreate() for swpctrie_zone and swblk_zone is
redundant, because uma_zone_reserve_kva() is performed on both zones and it
sets this same flag on the zone. (Moreover, the implementation of the swap
pager does not itself require these zones to be UMA_ZONE_NOFREE.)
Reviewed by: kib, markj
Approved by: re (gjb)
MFC after:1 week
Differential Revision:https://reviews.freebsd.org/D17296
Modified:
head/sys/vm/swap_pager.c
Modified: head/sys/vm/swap_pager.c
==
--- head/sys/vm/swap_pager.cMon Sep 24 16:48:54 2018(r338911)
+++ head/sys/vm/swap_pager.cMon Sep 24 16:49:02 2018(r338912)
@@ -545,13 +545,11 @@ swap_pager_swap_init(void)
if (maxswzone && n > maxswzone / sizeof(struct swblk))
n = maxswzone / sizeof(struct swblk);
swpctrie_zone = uma_zcreate("swpctrie", pctrie_node_size(), NULL, NULL,
- pctrie_zone_init, NULL, UMA_ALIGN_PTR,
- UMA_ZONE_NOFREE | UMA_ZONE_VM);
+ pctrie_zone_init, NULL, UMA_ALIGN_PTR, UMA_ZONE_VM);
if (swpctrie_zone == NULL)
panic("failed to create swap pctrie zone.");
swblk_zone = uma_zcreate("swblk", sizeof(struct swblk), NULL, NULL,
- NULL, NULL, _Alignof(struct swblk) - 1,
- UMA_ZONE_NOFREE | UMA_ZONE_VM);
+ NULL, NULL, _Alignof(struct swblk) - 1, UMA_ZONE_VM);
if (swblk_zone == NULL)
panic("failed to create swap blk zone.");
n2 = n;
___
svn-src-all@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-all
To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"
svn commit: r338431 - head/sys/vm
Author: alc
Date: Sun Sep 2 18:29:38 2018
New Revision: 338431
URL: https://svnweb.freebsd.org/changeset/base/338431
Log:
Recent changes have created, for the first time, physical memory segments
that can be coalesced. To be clear, fragmentation of phys_avail[] is not
the cause. This fragmentation of vm_phys_segs[] arises from the "special"
calls to vm_phys_add_seg(), in other words, not those that derive directly
from phys_avail[], but those that we create for the initial kernel page
table pages and now for the kernel and modules loaded at boot time. Since
we sometimes iterate over the physical memory segments, coalescing these
segments at initialization time is a worthwhile change.
Reviewed by: kib, markj
Approved by: re (rgrimes)
MFC after:1 week
Differential Revision:https://reviews.freebsd.org/D16976
Modified:
head/sys/vm/vm_phys.c
Modified: head/sys/vm/vm_phys.c
==
--- head/sys/vm/vm_phys.c Sun Sep 2 17:02:13 2018(r338430)
+++ head/sys/vm/vm_phys.c Sun Sep 2 18:29:38 2018(r338431)
@@ -460,7 +460,7 @@ void
vm_phys_init(void)
{
struct vm_freelist *fl;
- struct vm_phys_seg *seg;
+ struct vm_phys_seg *end_seg, *prev_seg, *seg, *tmp_seg;
u_long npages;
int dom, flind, freelist, oind, pind, segind;
@@ -544,6 +544,29 @@ vm_phys_init(void)
("vm_phys_init: DEFAULT flind < 0"));
}
seg->free_queues = _phys_free_queues[seg->domain][flind];
+ }
+
+ /*
+* Coalesce physical memory segments that are contiguous and share the
+* same per-domain free queues.
+*/
+ prev_seg = vm_phys_segs;
+ seg = _phys_segs[1];
+ end_seg = _phys_segs[vm_phys_nsegs];
+ while (seg < end_seg) {
+ if (prev_seg->end == seg->start &&
+ prev_seg->free_queues == seg->free_queues) {
+ prev_seg->end = seg->end;
+ KASSERT(prev_seg->domain == seg->domain,
+ ("vm_phys_init: free queues cannot span domains"));
+ vm_phys_nsegs--;
+ end_seg--;
+ for (tmp_seg = seg; tmp_seg < end_seg; tmp_seg++)
+ *tmp_seg = *(tmp_seg + 1);
+ } else {
+ prev_seg = seg;
+ seg++;
+ }
}
/*
___
svn-src-all@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-all
To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"
svn commit: r338318 - in head/sys: amd64/amd64 arm/allwinner arm/arm arm/freescale/imx arm/nvidia arm64/arm64 compat/linuxkpi/common/include/linux compat/linuxkpi/common/src dev/agp dev/amd_ecc_inj...
Author: alc
Date: Sat Aug 25 19:38:08 2018
New Revision: 338318
URL: https://svnweb.freebsd.org/changeset/base/338318
Log:
Eliminate the arena parameter to kmem_free(). Implicitly this corrects an
error in the function hypercall_memfree(), where the wrong arena was being
passed to kmem_free().
Introduce a per-page flag, VPO_KMEM_EXEC, to mark physical pages that are
mapped in kmem with execute permissions. Use this flag to determine which
arena the kmem virtual addresses are returned to.
Eliminate UMA_SLAB_KRWX. The introduction of VPO_KMEM_EXEC makes it
redundant.
Update the nearby comment for UMA_SLAB_KERNEL.
Reviewed by: kib, markj
Discussed with: jeff
Approved by: re (marius)
Differential Revision:https://reviews.freebsd.org/D16845
Modified:
head/sys/amd64/amd64/sys_machdep.c
head/sys/amd64/amd64/vm_machdep.c
head/sys/arm/allwinner/a10_fb.c
head/sys/arm/arm/busdma_machdep-v4.c
head/sys/arm/arm/busdma_machdep-v6.c
head/sys/arm/arm/pmap-v6.c
head/sys/arm/freescale/imx/imx6_sdma.c
head/sys/arm/nvidia/tegra_xhci.c
head/sys/arm64/arm64/busdma_bounce.c
head/sys/arm64/arm64/mp_machdep.c
head/sys/compat/linuxkpi/common/include/linux/dma-mapping.h
head/sys/compat/linuxkpi/common/src/linux_page.c
head/sys/dev/agp/agp.c
head/sys/dev/agp/agp_amd.c
head/sys/dev/agp/agp_ati.c
head/sys/dev/agp/agp_i810.c
head/sys/dev/amd_ecc_inject/ecc_inject.c
head/sys/dev/drm/drm_scatter.c
head/sys/dev/drm2/drm_scatter.c
head/sys/dev/hyperv/vmbus/hyperv.c
head/sys/dev/liquidio/lio_network.h
head/sys/kern/kern_malloc.c
head/sys/kern/subr_busdma_bufalloc.c
head/sys/mips/ingenic/jz4780_lcd.c
head/sys/mips/mips/busdma_machdep.c
head/sys/powerpc/powerpc/busdma_machdep.c
head/sys/vm/uma.h
head/sys/vm/uma_core.c
head/sys/vm/vm_extern.h
head/sys/vm/vm_kern.c
head/sys/vm/vm_page.h
head/sys/x86/iommu/busdma_dmar.c
head/sys/x86/iommu/intel_intrmap.c
head/sys/x86/iommu/intel_qi.c
head/sys/x86/x86/busdma_bounce.c
Modified: head/sys/amd64/amd64/sys_machdep.c
==
--- head/sys/amd64/amd64/sys_machdep.c Sat Aug 25 16:54:37 2018
(r338317)
+++ head/sys/amd64/amd64/sys_machdep.c Sat Aug 25 19:38:08 2018
(r338318)
@@ -479,7 +479,7 @@ user_ldt_alloc(struct proc *p, int force)
pldt = mdp->md_ldt;
if (pldt != NULL && !force) {
pmap_pti_remove_kva(sva, sva + sz);
- kmem_free(kernel_arena, sva, sz);
+ kmem_free(sva, sz);
free(new_ldt, M_SUBPROC);
return (pldt);
}
@@ -533,7 +533,7 @@ user_ldt_derefl(struct proc_ldt *pldt)
sva = (vm_offset_t)pldt->ldt_base;
sz = max_ldt_segment * sizeof(struct user_segment_descriptor);
pmap_pti_remove_kva(sva, sva + sz);
- kmem_free(kernel_arena, sva, sz);
+ kmem_free(sva, sz);
free(pldt, M_SUBPROC);
}
}
Modified: head/sys/amd64/amd64/vm_machdep.c
==
--- head/sys/amd64/amd64/vm_machdep.c Sat Aug 25 16:54:37 2018
(r338317)
+++ head/sys/amd64/amd64/vm_machdep.c Sat Aug 25 19:38:08 2018
(r338318)
@@ -331,8 +331,7 @@ cpu_thread_clean(struct thread *td)
if (pcb->pcb_tssp != NULL) {
pmap_pti_remove_kva((vm_offset_t)pcb->pcb_tssp,
(vm_offset_t)pcb->pcb_tssp + ctob(IOPAGES + 1));
- kmem_free(kernel_arena, (vm_offset_t)pcb->pcb_tssp,
- ctob(IOPAGES + 1));
+ kmem_free((vm_offset_t)pcb->pcb_tssp, ctob(IOPAGES + 1));
pcb->pcb_tssp = NULL;
}
}
Modified: head/sys/arm/allwinner/a10_fb.c
==
--- head/sys/arm/allwinner/a10_fb.c Sat Aug 25 16:54:37 2018
(r338317)
+++ head/sys/arm/allwinner/a10_fb.c Sat Aug 25 19:38:08 2018
(r338318)
@@ -192,7 +192,7 @@ a10fb_allocfb(struct a10fb_softc *sc)
static void
a10fb_freefb(struct a10fb_softc *sc)
{
- kmem_free(kernel_arena, sc->vaddr, sc->fbsize);
+ kmem_free(sc->vaddr, sc->fbsize);
}
static int
Modified: head/sys/arm/arm/busdma_machdep-v4.c
==
--- head/sys/arm/arm/busdma_machdep-v4.cSat Aug 25 16:54:37 2018
(r338317)
+++ head/sys/arm/arm/busdma_machdep-v4.cSat Aug 25 19:38:08 2018
(r338318)
@@ -792,7 +792,7 @@ bus_dmamem_free(bus_dma_tag_t dmat, void *vaddr, bus_d
!_bus_dma_can_bounce(dmat->lowaddr, dmat->highaddr))
uma_zfree(bufzone->umazone, vaddr);
else
- kmem_free(kernel_arena, (vm_offset_t)vaddr, dmat->maxsize);
+ kmem_free((vm_offset_t)vaddr, dmat->maxsize);
svn commit: r338143 - in head/sys: amd64/amd64 arm/arm arm64/arm64 compat/linuxkpi/common/src dev/hyperv/vmbus i386/i386 mips/mips powerpc/powerpc riscv/riscv sparc64/sparc64 vm x86/xen
Author: alc
Date: Tue Aug 21 16:43:46 2018
New Revision: 338143
URL: https://svnweb.freebsd.org/changeset/base/338143
Log:
Eliminate kmem_malloc()'s unused arena parameter. (The arena parameter
became unused in FreeBSD 12.x as a side-effect of the NUMA-related
changes.)
Reviewed by: kib, markj
Discussed with: jeff, re@
Differential Revision:https://reviews.freebsd.org/D16825
Modified:
head/sys/amd64/amd64/mp_machdep.c
head/sys/amd64/amd64/pmap.c
head/sys/amd64/amd64/sys_machdep.c
head/sys/arm/arm/mp_machdep.c
head/sys/arm/arm/pmap-v6.c
head/sys/arm64/arm64/mp_machdep.c
head/sys/arm64/arm64/pmap.c
head/sys/compat/linuxkpi/common/src/linux_page.c
head/sys/dev/hyperv/vmbus/hyperv.c
head/sys/i386/i386/mp_machdep.c
head/sys/i386/i386/pmap.c
head/sys/mips/mips/mp_machdep.c
head/sys/powerpc/powerpc/mp_machdep.c
head/sys/riscv/riscv/mp_machdep.c
head/sys/sparc64/sparc64/mp_machdep.c
head/sys/vm/uma_core.c
head/sys/vm/vm_extern.h
head/sys/vm/vm_init.c
head/sys/vm/vm_kern.c
head/sys/x86/xen/pv.c
Modified: head/sys/amd64/amd64/mp_machdep.c
==
--- head/sys/amd64/amd64/mp_machdep.c Tue Aug 21 16:37:37 2018
(r338142)
+++ head/sys/amd64/amd64/mp_machdep.c Tue Aug 21 16:43:46 2018
(r338143)
@@ -402,18 +402,14 @@ native_start_all_aps(void)
apic_id = cpu_apic_ids[cpu];
/* allocate and set up an idle stack data page */
- bootstacks[cpu] = (void *)kmem_malloc(kernel_arena,
- kstack_pages * PAGE_SIZE, M_WAITOK | M_ZERO);
- doublefault_stack = (char *)kmem_malloc(kernel_arena,
- PAGE_SIZE, M_WAITOK | M_ZERO);
- mce_stack = (char *)kmem_malloc(kernel_arena, PAGE_SIZE,
+ bootstacks[cpu] = (void *)kmem_malloc(kstack_pages * PAGE_SIZE,
M_WAITOK | M_ZERO);
- nmi_stack = (char *)kmem_malloc(kernel_arena, PAGE_SIZE,
- M_WAITOK | M_ZERO);
- dbg_stack = (char *)kmem_malloc(kernel_arena, PAGE_SIZE,
- M_WAITOK | M_ZERO);
- dpcpu = (void *)kmem_malloc(kernel_arena, DPCPU_SIZE,
- M_WAITOK | M_ZERO);
+ doublefault_stack = (char *)kmem_malloc(PAGE_SIZE, M_WAITOK |
+ M_ZERO);
+ mce_stack = (char *)kmem_malloc(PAGE_SIZE, M_WAITOK | M_ZERO);
+ nmi_stack = (char *)kmem_malloc(PAGE_SIZE, M_WAITOK | M_ZERO);
+ dbg_stack = (char *)kmem_malloc(PAGE_SIZE, M_WAITOK | M_ZERO);
+ dpcpu = (void *)kmem_malloc(DPCPU_SIZE, M_WAITOK | M_ZERO);
bootSTK = (char *)bootstacks[cpu] + kstack_pages * PAGE_SIZE -
8;
bootAP = cpu;
Modified: head/sys/amd64/amd64/pmap.c
==
--- head/sys/amd64/amd64/pmap.c Tue Aug 21 16:37:37 2018(r338142)
+++ head/sys/amd64/amd64/pmap.c Tue Aug 21 16:43:46 2018(r338143)
@@ -1412,8 +1412,7 @@ pmap_init(void)
*/
s = (vm_size_t)(pv_npg * sizeof(struct md_page));
s = round_page(s);
- pv_table = (struct md_page *)kmem_malloc(kernel_arena, s,
- M_WAITOK | M_ZERO);
+ pv_table = (struct md_page *)kmem_malloc(s, M_WAITOK | M_ZERO);
for (i = 0; i < pv_npg; i++)
TAILQ_INIT(_table[i].pv_list);
TAILQ_INIT(_dummy.pv_list);
Modified: head/sys/amd64/amd64/sys_machdep.c
==
--- head/sys/amd64/amd64/sys_machdep.c Tue Aug 21 16:37:37 2018
(r338142)
+++ head/sys/amd64/amd64/sys_machdep.c Tue Aug 21 16:43:46 2018
(r338143)
@@ -361,8 +361,8 @@ amd64_set_ioperm(td, uap)
*/
pcb = td->td_pcb;
if (pcb->pcb_tssp == NULL) {
- tssp = (struct amd64tss *)kmem_malloc(kernel_arena,
- ctob(IOPAGES + 1), M_WAITOK);
+ tssp = (struct amd64tss *)kmem_malloc(ctob(IOPAGES + 1),
+ M_WAITOK);
pmap_pti_add_kva((vm_offset_t)tssp, (vm_offset_t)tssp +
ctob(IOPAGES + 1), false);
iomap = (char *)[1];
@@ -463,7 +463,7 @@ user_ldt_alloc(struct proc *p, int force)
mtx_unlock(_lock);
new_ldt = malloc(sizeof(struct proc_ldt), M_SUBPROC, M_WAITOK);
sz = max_ldt_segment * sizeof(struct user_segment_descriptor);
- sva = kmem_malloc(kernel_arena, sz, M_WAITOK | M_ZERO);
+ sva = kmem_malloc(sz, M_WAITOK | M_ZERO);
new_ldt->ldt_base = (caddr_t)sva;
pmap_pti_add_kva(sva, sva + sz, false);
new_ldt->ldt_refcnt = 1;
Modified: head/sys/arm/arm/mp_machdep.c
==
--- head/sys/arm/arm/mp_machdep.c Tue Aug 21 16:37:37 2018
(r338142)
Re: svn commit: r338018 - head/sys/vm
On 08/20/2018 13:36, O. Hartmann wrote:
> Am Sat, 18 Aug 2018 18:33:50 + (UTC)
> Alan Cox schrieb:
>
> > Author: alc
> > Date: Sat Aug 18 18:33:50 2018
> > New Revision: 338018
> > URL: https://svnweb.freebsd.org/changeset/base/338018
>
> > Log:
> > Eliminate the arena parameter to kmem_malloc_domain(). It is
> redundant.
> > The domain and flags parameters suffice. In fact, the related
> functions
> > kmem_alloc_{attr,contig}_domain() don't have an arena parameter.
>
> > Reviewed by:kib, markj
> > Differential Revision:https://reviews.freebsd.org/D16713
>
> > Modified:
> > head/sys/vm/uma_core.c
> > head/sys/vm/vm_extern.h
> > head/sys/vm/vm_kern.c
>
> > Modified: head/sys/vm/uma_core.c
> >
> ==
> > --- head/sys/vm/uma_core.cSat Aug 18 16:03:15 2018(r338017)
> > +++ head/sys/vm/uma_core.cSat Aug 18 18:33:50 2018(r338018)
> > @@ -1169,7 +1169,7 @@ page_alloc(uma_zone_t zone, vm_size_t bytes,
> int domai
> > void *p;/* Returned page */
>
> > *pflag = UMA_SLAB_KERNEL;
> > -p = (void *) kmem_malloc_domain(kernel_arena, domain, bytes, wait);
> > +p = (void *) kmem_malloc_domain(domain, bytes, wait);
>
> > return (p);
> > }
> > @@ -3680,32 +3680,22 @@ uma_zone_exhausted_nolock(uma_zone_t zone)
> > void *
> > uma_large_malloc_domain(vm_size_t size, int domain, int wait)
> > {
> > -struct vmem *arena;
> > vm_offset_t addr;
> > uma_slab_t slab;
>
> > -#if VM_NRESERVLEVEL > 0
> > -if (__predict_true((wait & M_EXEC) == 0))
> > -arena = kernel_arena;
> > -else
> > -arena = kernel_rwx_arena;
> > -#else
> > -arena = kernel_arena;
> > -#endif
> > -
> > slab = zone_alloc_item(slabzone, NULL, domain, wait);
> > if (slab == NULL)
> > return (NULL);
> > if (domain == UMA_ANYDOMAIN)
> > -addr = kmem_malloc(arena, size, wait);
> > +addr = kmem_malloc(NULL, size, wait);
> > else
> > -addr = kmem_malloc_domain(arena, domain, size, wait);
> > +addr = kmem_malloc_domain(domain, size, wait);
> > if (addr != 0) {
> > vsetslab(addr, slab);
> > slab->us_data = (void *)addr;
> > slab->us_flags = UMA_SLAB_KERNEL | UMA_SLAB_MALLOC;
> > #if VM_NRESERVLEVEL > 0
> > -if (__predict_false(arena == kernel_rwx_arena))
> > +if (__predict_false((wait & M_EXEC) != 0))
> > slab->us_flags |= UMA_SLAB_KRWX;
> > #endif
> > slab->us_size = size;
>
> > Modified: head/sys/vm/vm_extern.h
> >
> ==
> > --- head/sys/vm/vm_extern.hSat Aug 18 16:03:15 2018(r338017)
> > +++ head/sys/vm/vm_extern.hSat Aug 18 18:33:50 2018(r338018)
> > @@ -65,8 +65,7 @@ vm_offset_t kmem_alloc_contig_domain(int domain, vm_si
> > vm_paddr_t low, vm_paddr_t high, u_long alignment, vm_paddr_t
> boundary,
> > vm_memattr_t memattr);
> > vm_offset_t kmem_malloc(struct vmem *, vm_size_t size, int flags);
> > -vm_offset_t kmem_malloc_domain(struct vmem *, int domain, vm_size_t
> size,
> > -int flags);
> > +vm_offset_t kmem_malloc_domain(int domain, vm_size_t size, int flags);
> > void kmem_free(struct vmem *, vm_offset_t, vm_size_t);
>
> > /* This provides memory for previously allocated address space. */
>
> > Modified: head/sys/vm/vm_kern.c
> >
> ==
> > --- head/sys/vm/vm_kern.cSat Aug 18 16:03:15 2018(r338017)
> > +++ head/sys/vm/vm_kern.cSat Aug 18 18:33:50 2018(r338018)
> > @@ -372,23 +372,18 @@ kmem_suballoc(vm_map_t parent, vm_offset_t
> *min, vm_of
> > *Allocate wired-down pages in the kernel's address space.
> > */
> > vm_offset_t
> > -kmem_malloc_domain(struct vmem *vmem, int domain, vm_size_t size,
> int flags)
> > +kmem_malloc_domain(int domain, vm_size_t size, int flags)
> > {
> > vmem_t *arena;
> > vm_offset_t addr;
> > int rv;
>
> > #if VM_NRESERVLEVEL > 0
> > -KASSERT(vmem == kernel_arena || vmem == kernel_rwx_arena,
> > -("kmem_malloc_domain: Only kernel_arena or kernel_rwx_arena "
> > -"are supported."));
> > -if (__predict_true(vmem == k
svn commit: r338107 - in head/sys: arm/allwinner arm/arm arm/freescale/imx arm/nvidia arm/nvidia/drm2 arm/samsung/exynos arm64/arm64 compat/linuxkpi/common/include/linux compat/linuxkpi/common/src ...
Author: alc
Date: Mon Aug 20 15:57:27 2018
New Revision: 338107
URL: https://svnweb.freebsd.org/changeset/base/338107
Log:
Eliminate kmem_alloc_contig()'s unused arena parameter.
Reviewed by: hselasky, kib, markj
Discussed with: jeff
Differential Revision:https://reviews.freebsd.org/D16799
Modified:
head/sys/arm/allwinner/a10_fb.c
head/sys/arm/arm/busdma_machdep-v4.c
head/sys/arm/arm/busdma_machdep-v6.c
head/sys/arm/arm/pmap-v6.c
head/sys/arm/freescale/imx/imx6_sdma.c
head/sys/arm/nvidia/drm2/tegra_dc.c
head/sys/arm/nvidia/tegra_pcie.c
head/sys/arm/nvidia/tegra_xhci.c
head/sys/arm/samsung/exynos/exynos5_fimd.c
head/sys/arm64/arm64/busdma_bounce.c
head/sys/compat/linuxkpi/common/include/linux/dma-mapping.h
head/sys/compat/linuxkpi/common/src/linux_page.c
head/sys/compat/ndis/subr_ntoskrnl.c
head/sys/dev/agp/agp.c
head/sys/dev/agp/agp_i810.c
head/sys/dev/liquidio/lio_network.h
head/sys/dev/xdma/controller/pl330.c
head/sys/kern/kern_malloc.c
head/sys/mips/ingenic/jz4780_lcd.c
head/sys/mips/mips/busdma_machdep.c
head/sys/powerpc/powerpc/busdma_machdep.c
head/sys/vm/vm_extern.h
head/sys/vm/vm_kern.c
head/sys/x86/iommu/intel_intrmap.c
head/sys/x86/iommu/intel_qi.c
Modified: head/sys/arm/allwinner/a10_fb.c
==
--- head/sys/arm/allwinner/a10_fb.c Mon Aug 20 15:43:08 2018
(r338106)
+++ head/sys/arm/allwinner/a10_fb.c Mon Aug 20 15:57:27 2018
(r338107)
@@ -178,8 +178,8 @@ static struct resource_spec a10fb_spec[] = {
static int
a10fb_allocfb(struct a10fb_softc *sc)
{
- sc->vaddr = kmem_alloc_contig(kernel_arena, sc->fbsize,
- M_NOWAIT | M_ZERO, 0, ~0, FB_ALIGN, 0, VM_MEMATTR_WRITE_COMBINING);
+ sc->vaddr = kmem_alloc_contig(sc->fbsize, M_NOWAIT | M_ZERO, 0, ~0,
+ FB_ALIGN, 0, VM_MEMATTR_WRITE_COMBINING);
if (sc->vaddr == 0) {
device_printf(sc->dev, "failed to allocate FB memory\n");
return (ENOMEM);
Modified: head/sys/arm/arm/busdma_machdep-v4.c
==
--- head/sys/arm/arm/busdma_machdep-v4.cMon Aug 20 15:43:08 2018
(r338106)
+++ head/sys/arm/arm/busdma_machdep-v4.cMon Aug 20 15:57:27 2018
(r338107)
@@ -750,9 +750,8 @@ bus_dmamem_alloc(bus_dma_tag_t dmat, void **vaddr, int
*vaddr = (void *)kmem_alloc_attr(dmat->maxsize, mflags, 0,
dmat->lowaddr, memattr);
} else {
- *vaddr = (void *)kmem_alloc_contig(kernel_arena, dmat->maxsize,
- mflags, 0, dmat->lowaddr, dmat->alignment, dmat->boundary,
- memattr);
+ *vaddr = (void *)kmem_alloc_contig(dmat->maxsize, mflags, 0,
+ dmat->lowaddr, dmat->alignment, dmat->boundary, memattr);
}
if (*vaddr == NULL) {
CTR4(KTR_BUSDMA, "%s: tag %p tag flags 0x%x error %d",
Modified: head/sys/arm/arm/busdma_machdep-v6.c
==
--- head/sys/arm/arm/busdma_machdep-v6.cMon Aug 20 15:43:08 2018
(r338106)
+++ head/sys/arm/arm/busdma_machdep-v6.cMon Aug 20 15:57:27 2018
(r338107)
@@ -815,9 +815,8 @@ bus_dmamem_alloc(bus_dma_tag_t dmat, void **vaddr, int
*vaddr = (void *)kmem_alloc_attr(dmat->maxsize, mflags, 0,
dmat->lowaddr, memattr);
} else {
- *vaddr = (void *)kmem_alloc_contig(kernel_arena, dmat->maxsize,
- mflags, 0, dmat->lowaddr, dmat->alignment, dmat->boundary,
- memattr);
+ *vaddr = (void *)kmem_alloc_contig(dmat->maxsize, mflags, 0,
+ dmat->lowaddr, dmat->alignment, dmat->boundary, memattr);
}
if (*vaddr == NULL) {
CTR4(KTR_BUSDMA, "%s: tag %p tag flags 0x%x error %d",
Modified: head/sys/arm/arm/pmap-v6.c
==
--- head/sys/arm/arm/pmap-v6.c Mon Aug 20 15:43:08 2018(r338106)
+++ head/sys/arm/arm/pmap-v6.c Mon Aug 20 15:57:27 2018(r338107)
@@ -2219,9 +2219,8 @@ pmap_pinit(pmap_t pmap)
*/
if (pmap->pm_pt1 == NULL) {
- pmap->pm_pt1 = (pt1_entry_t *)kmem_alloc_contig(kernel_arena,
- NB_IN_PT1, M_NOWAIT | M_ZERO, 0, -1UL, NB_IN_PT1, 0,
- pt_memattr);
+ pmap->pm_pt1 = (pt1_entry_t *)kmem_alloc_contig(NB_IN_PT1,
+ M_NOWAIT | M_ZERO, 0, -1UL, NB_IN_PT1, 0, pt_memattr);
if (pmap->pm_pt1 == NULL)
return (0);
}
Modified: head/sys/arm/freescale/imx/imx6_sdma.c
==
--- head/sys/arm/freescale/imx/imx6_sdma.c Mon Aug 20
svn commit: r338032 - in head/sys: arm/arm arm64/arm64 dev/agp mips/mips
Author: alc
Date: Sat Aug 18 22:35:19 2018
New Revision: 338032
URL: https://svnweb.freebsd.org/changeset/base/338032
Log:
Oops. r338030 didn't eliminate the unused arena argument from all of
kmem_alloc_attr()'s callers. Correct that mistake.
Modified:
head/sys/arm/arm/busdma_machdep-v4.c
head/sys/arm/arm/busdma_machdep-v6.c
head/sys/arm/arm/pmap-v6.c
head/sys/arm64/arm64/busdma_bounce.c
head/sys/dev/agp/agp_amd.c
head/sys/dev/agp/agp_ati.c
head/sys/mips/mips/busdma_machdep.c
Modified: head/sys/arm/arm/busdma_machdep-v4.c
==
--- head/sys/arm/arm/busdma_machdep-v4.cSat Aug 18 22:21:59 2018
(r338031)
+++ head/sys/arm/arm/busdma_machdep-v4.cSat Aug 18 22:35:19 2018
(r338032)
@@ -747,8 +747,8 @@ bus_dmamem_alloc(bus_dma_tag_t dmat, void **vaddr, int
howmany(dmat->maxsize, MIN(dmat->maxsegsz, PAGE_SIZE)) &&
dmat->alignment <= PAGE_SIZE &&
(dmat->boundary % PAGE_SIZE) == 0) {
- *vaddr = (void *)kmem_alloc_attr(kernel_arena, dmat->maxsize,
- mflags, 0, dmat->lowaddr, memattr);
+ *vaddr = (void *)kmem_alloc_attr(dmat->maxsize, mflags, 0,
+ dmat->lowaddr, memattr);
} else {
*vaddr = (void *)kmem_alloc_contig(kernel_arena, dmat->maxsize,
mflags, 0, dmat->lowaddr, dmat->alignment, dmat->boundary,
Modified: head/sys/arm/arm/busdma_machdep-v6.c
==
--- head/sys/arm/arm/busdma_machdep-v6.cSat Aug 18 22:21:59 2018
(r338031)
+++ head/sys/arm/arm/busdma_machdep-v6.cSat Aug 18 22:35:19 2018
(r338032)
@@ -812,8 +812,8 @@ bus_dmamem_alloc(bus_dma_tag_t dmat, void **vaddr, int
howmany(dmat->maxsize, MIN(dmat->maxsegsz, PAGE_SIZE)) &&
dmat->alignment <= PAGE_SIZE &&
(dmat->boundary % PAGE_SIZE) == 0) {
- *vaddr = (void *)kmem_alloc_attr(kernel_arena, dmat->maxsize,
- mflags, 0, dmat->lowaddr, memattr);
+ *vaddr = (void *)kmem_alloc_attr(dmat->maxsize, mflags, 0,
+ dmat->lowaddr, memattr);
} else {
*vaddr = (void *)kmem_alloc_contig(kernel_arena, dmat->maxsize,
mflags, 0, dmat->lowaddr, dmat->alignment, dmat->boundary,
Modified: head/sys/arm/arm/pmap-v6.c
==
--- head/sys/arm/arm/pmap-v6.c Sat Aug 18 22:21:59 2018(r338031)
+++ head/sys/arm/arm/pmap-v6.c Sat Aug 18 22:35:19 2018(r338032)
@@ -2236,8 +2236,8 @@ pmap_pinit(pmap_t pmap)
* be used no matter which process is current. Its mapping
* in PT2MAP can be used only for current process.
*/
- pmap->pm_pt2tab = (pt2_entry_t *)kmem_alloc_attr(kernel_arena,
- NB_IN_PT2TAB, M_NOWAIT | M_ZERO, 0, -1UL, pt_memattr);
+ pmap->pm_pt2tab = (pt2_entry_t *)kmem_alloc_attr(NB_IN_PT2TAB,
+ M_NOWAIT | M_ZERO, 0, -1UL, pt_memattr);
if (pmap->pm_pt2tab == NULL) {
/*
* QQQ: As struct pmap is allocated from UMA with
Modified: head/sys/arm64/arm64/busdma_bounce.c
==
--- head/sys/arm64/arm64/busdma_bounce.cSat Aug 18 22:21:59 2018
(r338031)
+++ head/sys/arm64/arm64/busdma_bounce.cSat Aug 18 22:35:19 2018
(r338032)
@@ -491,9 +491,8 @@ bounce_bus_dmamem_alloc(bus_dma_tag_t dmat, void** vad
dmat->common.alignment <= PAGE_SIZE &&
(dmat->common.boundary % PAGE_SIZE) == 0) {
/* Page-based multi-segment allocations allowed */
- *vaddr = (void *)kmem_alloc_attr(kernel_arena,
- dmat->common.maxsize, mflags, 0ul, dmat->common.lowaddr,
- attr);
+ *vaddr = (void *)kmem_alloc_attr(dmat->common.maxsize, mflags,
+ 0ul, dmat->common.lowaddr, attr);
dmat->bounce_flags |= BF_KMEM_ALLOC;
} else {
*vaddr = (void *)kmem_alloc_contig(kernel_arena,
Modified: head/sys/dev/agp/agp_amd.c
==
--- head/sys/dev/agp/agp_amd.c Sat Aug 18 22:21:59 2018(r338031)
+++ head/sys/dev/agp/agp_amd.c Sat Aug 18 22:35:19 2018(r338032)
@@ -101,9 +101,8 @@ agp_amd_alloc_gatt(device_t dev)
* directory.
*/
gatt->ag_entries = entries;
- gatt->ag_virtual = (void *)kmem_alloc_attr(kernel_arena,
- entries * sizeof(u_int32_t), M_NOWAIT | M_ZERO, 0, ~0,
- VM_MEMATTR_WRITE_COMBINING);
+ gatt->ag_virtual = (void *)kmem_alloc_attr(entries *
svn commit: r338030 - in head/sys: dev/amd_ecc_inject dev/drm dev/drm2 vm
Author: alc
Date: Sat Aug 18 22:07:48 2018
New Revision: 338030
URL: https://svnweb.freebsd.org/changeset/base/338030
Log:
Eliminate the unused arena parameter from kmem_alloc_attr().
Reviewed by: kib, markj
Differential Revision:https://reviews.freebsd.org/D16793
Modified:
head/sys/dev/amd_ecc_inject/ecc_inject.c
head/sys/dev/drm/drm_scatter.c
head/sys/dev/drm2/drm_scatter.c
head/sys/vm/vm_extern.h
head/sys/vm/vm_init.c
head/sys/vm/vm_kern.c
Modified: head/sys/dev/amd_ecc_inject/ecc_inject.c
==
--- head/sys/dev/amd_ecc_inject/ecc_inject.cSat Aug 18 21:21:06 2018
(r338029)
+++ head/sys/dev/amd_ecc_inject/ecc_inject.cSat Aug 18 22:07:48 2018
(r338030)
@@ -186,7 +186,7 @@ ecc_ei_inject(int count)
KASSERT(bit_mask != 0 && (bit_mask & ~INJ_VECTOR_MASK) == 0,
("bit mask value is outside of range: 0x%x", bit_mask));
- memory = kmem_alloc_attr(kernel_arena, PAGE_SIZE, M_WAITOK, 0, ~0,
+ memory = kmem_alloc_attr(PAGE_SIZE, M_WAITOK, 0, ~0,
VM_MEMATTR_UNCACHEABLE);
for (injected = 0; injected < count; injected++) {
Modified: head/sys/dev/drm/drm_scatter.c
==
--- head/sys/dev/drm/drm_scatter.c Sat Aug 18 21:21:06 2018
(r338029)
+++ head/sys/dev/drm/drm_scatter.c Sat Aug 18 22:07:48 2018
(r338030)
@@ -52,8 +52,8 @@ drm_sg_alloc(struct drm_device *dev, struct drm_scatte
entry->busaddr = malloc(entry->pages * sizeof(*entry->busaddr),
DRM_MEM_SGLISTS, M_WAITOK | M_ZERO);
- entry->vaddr = kmem_alloc_attr(kernel_arena, size, M_WAITOK | M_ZERO,
- 0, BUS_SPACE_MAXADDR_32BIT, VM_MEMATTR_WRITE_COMBINING);
+ entry->vaddr = kmem_alloc_attr(size, M_WAITOK | M_ZERO, 0,
+ BUS_SPACE_MAXADDR_32BIT, VM_MEMATTR_WRITE_COMBINING);
if (entry->vaddr == 0) {
drm_sg_cleanup(entry);
return (ENOMEM);
Modified: head/sys/dev/drm2/drm_scatter.c
==
--- head/sys/dev/drm2/drm_scatter.c Sat Aug 18 21:21:06 2018
(r338029)
+++ head/sys/dev/drm2/drm_scatter.c Sat Aug 18 22:07:48 2018
(r338030)
@@ -37,8 +37,8 @@ __FBSDID("$FreeBSD$");
static inline vm_offset_t drm_vmalloc_dma(vm_size_t size)
{
- return kmem_alloc_attr(kernel_arena, size, M_NOWAIT | M_ZERO,
- 0, BUS_SPACE_MAXADDR_32BIT, VM_MEMATTR_WRITE_COMBINING);
+ return kmem_alloc_attr(size, M_NOWAIT | M_ZERO, 0,
+ BUS_SPACE_MAXADDR_32BIT, VM_MEMATTR_WRITE_COMBINING);
}
void drm_sg_cleanup(struct drm_sg_mem * entry)
Modified: head/sys/vm/vm_extern.h
==
--- head/sys/vm/vm_extern.h Sat Aug 18 21:21:06 2018(r338029)
+++ head/sys/vm/vm_extern.h Sat Aug 18 22:07:48 2018(r338030)
@@ -54,7 +54,7 @@ vm_offset_t kmap_alloc_wait(vm_map_t, vm_size_t);
void kmap_free_wakeup(vm_map_t, vm_offset_t, vm_size_t);
/* These operate on virtual addresses backed by memory. */
-vm_offset_t kmem_alloc_attr(struct vmem *, vm_size_t size, int flags,
+vm_offset_t kmem_alloc_attr(vm_size_t size, int flags,
vm_paddr_t low, vm_paddr_t high, vm_memattr_t memattr);
vm_offset_t kmem_alloc_attr_domain(int domain, vm_size_t size, int flags,
vm_paddr_t low, vm_paddr_t high, vm_memattr_t memattr);
Modified: head/sys/vm/vm_init.c
==
--- head/sys/vm/vm_init.c Sat Aug 18 21:21:06 2018(r338029)
+++ head/sys/vm/vm_init.c Sat Aug 18 22:07:48 2018(r338030)
@@ -274,9 +274,8 @@ again:
* Try to protect 32-bit DMAable memory from the largest
* early alloc of wired mem.
*/
- firstaddr = kmem_alloc_attr(kernel_arena, size,
- M_ZERO | M_NOWAIT, (vm_paddr_t)1 << 32,
- ~(vm_paddr_t)0, VM_MEMATTR_DEFAULT);
+ firstaddr = kmem_alloc_attr(size, M_ZERO | M_NOWAIT,
+ (vm_paddr_t)1 << 32, ~(vm_paddr_t)0, VM_MEMATTR_DEFAULT);
if (firstaddr == 0)
#endif
firstaddr = kmem_malloc(kernel_arena, size,
Modified: head/sys/vm/vm_kern.c
==
--- head/sys/vm/vm_kern.c Sat Aug 18 21:21:06 2018(r338029)
+++ head/sys/vm/vm_kern.c Sat Aug 18 22:07:48 2018(r338030)
@@ -220,15 +220,12 @@ retry:
}
vm_offset_t
-kmem_alloc_attr(vmem_t *vmem, vm_size_t size, int flags, vm_paddr_t low,
-vm_paddr_t high, vm_memattr_t memattr)
+kmem_alloc_attr(vm_size_t size, int flags, vm_paddr_t low, vm_paddr_t high,
+vm_memattr_t memattr)
{
struct vm_domainset_iter di;
svn commit: r338018 - head/sys/vm
Author: alc
Date: Sat Aug 18 18:33:50 2018
New Revision: 338018
URL: https://svnweb.freebsd.org/changeset/base/338018
Log:
Eliminate the arena parameter to kmem_malloc_domain(). It is redundant.
The domain and flags parameters suffice. In fact, the related functions
kmem_alloc_{attr,contig}_domain() don't have an arena parameter.
Reviewed by: kib, markj
Differential Revision:https://reviews.freebsd.org/D16713
Modified:
head/sys/vm/uma_core.c
head/sys/vm/vm_extern.h
head/sys/vm/vm_kern.c
Modified: head/sys/vm/uma_core.c
==
--- head/sys/vm/uma_core.c Sat Aug 18 16:03:15 2018(r338017)
+++ head/sys/vm/uma_core.c Sat Aug 18 18:33:50 2018(r338018)
@@ -1169,7 +1169,7 @@ page_alloc(uma_zone_t zone, vm_size_t bytes, int domai
void *p;/* Returned page */
*pflag = UMA_SLAB_KERNEL;
- p = (void *) kmem_malloc_domain(kernel_arena, domain, bytes, wait);
+ p = (void *) kmem_malloc_domain(domain, bytes, wait);
return (p);
}
@@ -3680,32 +3680,22 @@ uma_zone_exhausted_nolock(uma_zone_t zone)
void *
uma_large_malloc_domain(vm_size_t size, int domain, int wait)
{
- struct vmem *arena;
vm_offset_t addr;
uma_slab_t slab;
-#if VM_NRESERVLEVEL > 0
- if (__predict_true((wait & M_EXEC) == 0))
- arena = kernel_arena;
- else
- arena = kernel_rwx_arena;
-#else
- arena = kernel_arena;
-#endif
-
slab = zone_alloc_item(slabzone, NULL, domain, wait);
if (slab == NULL)
return (NULL);
if (domain == UMA_ANYDOMAIN)
- addr = kmem_malloc(arena, size, wait);
+ addr = kmem_malloc(NULL, size, wait);
else
- addr = kmem_malloc_domain(arena, domain, size, wait);
+ addr = kmem_malloc_domain(domain, size, wait);
if (addr != 0) {
vsetslab(addr, slab);
slab->us_data = (void *)addr;
slab->us_flags = UMA_SLAB_KERNEL | UMA_SLAB_MALLOC;
#if VM_NRESERVLEVEL > 0
- if (__predict_false(arena == kernel_rwx_arena))
+ if (__predict_false((wait & M_EXEC) != 0))
slab->us_flags |= UMA_SLAB_KRWX;
#endif
slab->us_size = size;
Modified: head/sys/vm/vm_extern.h
==
--- head/sys/vm/vm_extern.h Sat Aug 18 16:03:15 2018(r338017)
+++ head/sys/vm/vm_extern.h Sat Aug 18 18:33:50 2018(r338018)
@@ -65,8 +65,7 @@ vm_offset_t kmem_alloc_contig_domain(int domain, vm_si
vm_paddr_t low, vm_paddr_t high, u_long alignment, vm_paddr_t boundary,
vm_memattr_t memattr);
vm_offset_t kmem_malloc(struct vmem *, vm_size_t size, int flags);
-vm_offset_t kmem_malloc_domain(struct vmem *, int domain, vm_size_t size,
-int flags);
+vm_offset_t kmem_malloc_domain(int domain, vm_size_t size, int flags);
void kmem_free(struct vmem *, vm_offset_t, vm_size_t);
/* This provides memory for previously allocated address space. */
Modified: head/sys/vm/vm_kern.c
==
--- head/sys/vm/vm_kern.c Sat Aug 18 16:03:15 2018(r338017)
+++ head/sys/vm/vm_kern.c Sat Aug 18 18:33:50 2018(r338018)
@@ -372,23 +372,18 @@ kmem_suballoc(vm_map_t parent, vm_offset_t *min, vm_of
* Allocate wired-down pages in the kernel's address space.
*/
vm_offset_t
-kmem_malloc_domain(struct vmem *vmem, int domain, vm_size_t size, int flags)
+kmem_malloc_domain(int domain, vm_size_t size, int flags)
{
vmem_t *arena;
vm_offset_t addr;
int rv;
#if VM_NRESERVLEVEL > 0
- KASSERT(vmem == kernel_arena || vmem == kernel_rwx_arena,
- ("kmem_malloc_domain: Only kernel_arena or kernel_rwx_arena "
- "are supported."));
- if (__predict_true(vmem == kernel_arena))
+ if (__predict_true((flags & M_EXEC) == 0))
arena = vm_dom[domain].vmd_kernel_arena;
else
arena = vm_dom[domain].vmd_kernel_rwx_arena;
#else
- KASSERT(vmem == kernel_arena,
- ("kmem_malloc_domain: Only kernel_arena is supported."));
arena = vm_dom[domain].vmd_kernel_arena;
#endif
size = round_page(size);
@@ -404,7 +399,7 @@ kmem_malloc_domain(struct vmem *vmem, int domain, vm_s
}
vm_offset_t
-kmem_malloc(struct vmem *vmem, vm_size_t size, int flags)
+kmem_malloc(struct vmem *vmem __unused, vm_size_t size, int flags)
{
struct vm_domainset_iter di;
vm_offset_t addr;
@@ -412,7 +407,7 @@ kmem_malloc(struct vmem *vmem, vm_size_t size, int fla
vm_domainset_iter_malloc_init(, kernel_object, , );
do {
- addr = kmem_malloc_domain(vmem, domain, size, flags);
+ addr = kmem_malloc_domain(domain,
svn commit: r337652 - head/sys/kern
Author: alc Date: Sat Aug 11 19:21:53 2018 New Revision: 337652 URL: https://svnweb.freebsd.org/changeset/base/337652 Log: Eliminate a redundant assignment. MFC after:1 week Modified: head/sys/kern/subr_vmem.c Modified: head/sys/kern/subr_vmem.c == --- head/sys/kern/subr_vmem.c Sat Aug 11 19:20:06 2018(r337651) +++ head/sys/kern/subr_vmem.c Sat Aug 11 19:21:53 2018(r337652) @@ -989,7 +989,6 @@ vmem_clip(vmem_t *vm, bt_t *bt, vmem_addr_t start, vme bt_insbusy(vm, bt); } MPASS(bt->bt_size >= size); - bt->bt_type = BT_TYPE_BUSY; } /* vmem API */ ___ svn-src-all@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"
svn commit: r337463 - in head/sys: arm/arm arm/include vm
Author: alc
Date: Wed Aug 8 16:55:01 2018
New Revision: 337463
URL: https://svnweb.freebsd.org/changeset/base/337463
Log:
Add support for pmap_enter(..., psind=1) to the armv6 pmap. In other words,
add support for explicitly requesting that pmap_enter() create a 1 MB page
mapping. (Essentially, this feature allows the machine-independent layer
to create superpage mappings preemptively, and not wait for automatic
promotion to occur.)
Export pmap_ps_enabled() to the machine-independent layer.
Add a flag to pmap_pv_insert_pte1() that specifies whether it should fail
or reclaim a PV entry when one is not available.
Refactor pmap_enter_pte1() into two functions, one by the same name, that
is a general-purpose function for creating pte1 mappings, and another,
pmap_enter_1mpage(), that is used to prefault 1 MB read- and/or execute-
only mappings for execve(2), mmap(2), and shmat(2).
In addition, as an optimization to pmap_enter(..., psind=0), eliminate the
use of pte2_is_managed() from pmap_enter(). Unlike the x86 pmap
implementations, armv6 does not have a managed bit defined within the PTE.
So, pte2_is_managed() is actually a call to PHYS_TO_VM_PAGE(), which is O(n)
in the number of vm_phys_segs[]. All but one call to PHYS_TO_VM_PAGE() in
pmap_enter() can be avoided.
Reviewed by: kib, markj, mmel
Tested by:mmel
MFC after:6 weeks
Differential Revision:https://reviews.freebsd.org/D16555
Modified:
head/sys/arm/arm/pmap-v6.c
head/sys/arm/include/pmap-v6.h
head/sys/vm/vm_fault.c
Modified: head/sys/arm/arm/pmap-v6.c
==
--- head/sys/arm/arm/pmap-v6.c Wed Aug 8 16:17:50 2018(r337462)
+++ head/sys/arm/arm/pmap-v6.c Wed Aug 8 16:55:01 2018(r337463)
@@ -323,9 +323,17 @@ SYSCTL_INT(_debug, OID_AUTO, PMAP1unchanged, CTLFLAG_R
"Number of times pmap_pte2_quick didn't change PMAP1");
static struct mtx PMAP2mutex;
+/*
+ * Internal flags for pmap_enter()'s helper functions.
+ */
+#definePMAP_ENTER_NORECLAIM0x100 /* Don't reclaim PV
entries. */
+#definePMAP_ENTER_NOREPLACE0x200 /* Don't replace
mappings. */
+
static __inline void pt2_wirecount_init(vm_page_t m);
static boolean_t pmap_demote_pte1(pmap_t pmap, pt1_entry_t *pte1p,
vm_offset_t va);
+static int pmap_enter_pte1(pmap_t pmap, vm_offset_t va, pt1_entry_t pte1,
+u_int flags, vm_page_t m);
void cache_icache_sync_fresh(vm_offset_t va, vm_paddr_t pa, vm_size_t size);
/*
@@ -1557,6 +1565,13 @@ static int sp_enabled = 1;
SYSCTL_INT(_vm_pmap, OID_AUTO, sp_enabled, CTLFLAG_RDTUN | CTLFLAG_NOFETCH,
_enabled, 0, "Are large page mappings enabled?");
+bool
+pmap_ps_enabled(pmap_t pmap __unused)
+{
+
+ return (sp_enabled != 0);
+}
+
static SYSCTL_NODE(_vm_pmap, OID_AUTO, pte1, CTLFLAG_RD, 0,
"1MB page mapping counters");
@@ -3227,21 +3242,22 @@ pmap_try_insert_pv_entry(pmap_t pmap, vm_offset_t va,
/*
* Create the pv entries for each of the pages within a section.
*/
-static boolean_t
-pmap_pv_insert_pte1(pmap_t pmap, vm_offset_t va, vm_paddr_t pa)
+static bool
+pmap_pv_insert_pte1(pmap_t pmap, vm_offset_t va, pt1_entry_t pte1, u_int flags)
{
struct md_page *pvh;
pv_entry_t pv;
+ bool noreclaim;
rw_assert(_global_lock, RA_WLOCKED);
- if (pv_entry_count < pv_entry_high_water &&
- (pv = get_pv_entry(pmap, TRUE)) != NULL) {
- pv->pv_va = va;
- pvh = pa_to_pvh(pa);
- TAILQ_INSERT_TAIL(>pv_list, pv, pv_next);
- return (TRUE);
- } else
- return (FALSE);
+ noreclaim = (flags & PMAP_ENTER_NORECLAIM) != 0;
+ if ((noreclaim && pv_entry_count >= pv_entry_high_water) ||
+ (pv = get_pv_entry(pmap, noreclaim)) == NULL)
+ return (false);
+ pv->pv_va = va;
+ pvh = pa_to_pvh(pte1_pa(pte1));
+ TAILQ_INSERT_TAIL(>pv_list, pv, pv_next);
+ return (true);
}
static inline void
@@ -3879,7 +3895,7 @@ pmap_enter(pmap_t pmap, vm_offset_t va, vm_page_t m, v
if ((prot & VM_PROT_WRITE) == 0)
npte2 |= PTE2_RO;
KASSERT((npte2 & (PTE2_NM | PTE2_RO)) != PTE2_RO,
- ("pmap_enter: flags includes VM_PROT_WRITE but prot doesn't"));
+ ("%s: flags includes VM_PROT_WRITE but prot doesn't", __func__));
if ((prot & VM_PROT_EXECUTE) == 0)
npte2 |= PTE2_NX;
if ((flags & PMAP_ENTER_WIRED) != 0)
@@ -3892,6 +3908,15 @@ pmap_enter(pmap_t pmap, vm_offset_t va, vm_page_t m, v
rw_wlock(_global_lock);
PMAP_LOCK(pmap);
sched_pin();
+ if (psind == 1) {
+ /* Assert the required virtual and physical alignment. */
+ KASSERT((va & PTE1_OFFSET) == 0,
+ ("%s: va unaligned", __func__));
+ KASSERT(m->psind > 0,
svn commit: r337443 - head/sys/vm
Author: alc
Date: Wed Aug 8 02:30:34 2018
New Revision: 337443
URL: https://svnweb.freebsd.org/changeset/base/337443
Log:
Defer and aggregate swap_pager_meta_build frees.
Before swp_pager_meta_build replaces an old swapblk with an new one,
it frees the old one. To allow such freeing of blocks to be
aggregated, have swp_pager_meta_build return the old swap block, and
make the caller responsible for freeing it.
Define a pair of short static functions, swp_pager_init_freerange and
swp_pager_update_freerange, to do the initialization and updating of
blk addresses and counters used in aggregating blocks to be freed.
Submitted by: Doug Moore
Reviewed by: kib, markj (an earlier version)
Tested by:pho
MFC after:1 week
Differential Revision:https://reviews.freebsd.org/D13707
Modified:
head/sys/vm/swap_pager.c
Modified: head/sys/vm/swap_pager.c
==
--- head/sys/vm/swap_pager.cWed Aug 8 01:33:36 2018(r337442)
+++ head/sys/vm/swap_pager.cWed Aug 8 02:30:34 2018(r337443)
@@ -403,11 +403,32 @@ static daddr_tswp_pager_getswapspace(int npages);
/*
* Metadata functions
*/
-static void swp_pager_meta_build(vm_object_t, vm_pindex_t, daddr_t);
+static daddr_t swp_pager_meta_build(vm_object_t, vm_pindex_t, daddr_t);
static void swp_pager_meta_free(vm_object_t, vm_pindex_t, vm_pindex_t);
static void swp_pager_meta_free_all(vm_object_t);
static daddr_t swp_pager_meta_ctl(vm_object_t, vm_pindex_t, int);
+static void
+swp_pager_init_freerange(daddr_t *start, daddr_t *num)
+{
+
+ *start = SWAPBLK_NONE;
+ *num = 0;
+}
+
+static void
+swp_pager_update_freerange(daddr_t *start, daddr_t *num, daddr_t addr)
+{
+
+ if (*start + *num == addr) {
+ (*num)++;
+ } else {
+ swp_pager_freeswapspace(*start, *num);
+ *start = addr;
+ *num = 1;
+ }
+}
+
static void *
swblk_trie_alloc(struct pctrie *ptree)
{
@@ -861,7 +882,9 @@ swap_pager_reserve(vm_object_t object, vm_pindex_t sta
int n = 0;
daddr_t blk = SWAPBLK_NONE;
vm_pindex_t beg = start;/* save start index */
+ daddr_t addr, n_free, s_free;
+ swp_pager_init_freerange(_free, _free);
VM_OBJECT_WLOCK(object);
while (size) {
if (n == 0) {
@@ -875,12 +898,15 @@ swap_pager_reserve(vm_object_t object, vm_pindex_t sta
}
}
}
- swp_pager_meta_build(object, start, blk);
+ addr = swp_pager_meta_build(object, start, blk);
+ if (addr != SWAPBLK_NONE)
+ swp_pager_update_freerange(_free, _free, addr);
--size;
++start;
++blk;
--n;
}
+ swp_pager_freeswapspace(s_free, n_free);
swp_pager_meta_free(object, start, n);
VM_OBJECT_WUNLOCK(object);
return (0);
@@ -910,7 +936,7 @@ swap_pager_copy(vm_object_t srcobject, vm_object_t dst
vm_pindex_t offset, int destroysource)
{
vm_pindex_t i;
- daddr_t dstaddr, first_free, num_free, srcaddr;
+ daddr_t dstaddr, n_free, s_free, srcaddr;
VM_OBJECT_ASSERT_WLOCKED(srcobject);
VM_OBJECT_ASSERT_WLOCKED(dstobject);
@@ -937,42 +963,38 @@ swap_pager_copy(vm_object_t srcobject, vm_object_t dst
/*
* Transfer source to destination.
*/
- first_free = SWAPBLK_NONE;
- num_free = 0;
+ swp_pager_init_freerange(_free, _free);
for (i = 0; i < dstobject->size; ++i) {
srcaddr = swp_pager_meta_ctl(srcobject, i + offset, SWM_POP);
if (srcaddr == SWAPBLK_NONE)
continue;
dstaddr = swp_pager_meta_ctl(dstobject, i, 0);
- if (dstaddr == SWAPBLK_NONE) {
+ if (dstaddr != SWAPBLK_NONE) {
/*
-* Destination has no swapblk and is not resident,
-* copy source.
-*
-* swp_pager_meta_build() can sleep.
-*/
- vm_object_pip_add(srcobject, 1);
- VM_OBJECT_WUNLOCK(srcobject);
- vm_object_pip_add(dstobject, 1);
- swp_pager_meta_build(dstobject, i, srcaddr);
- vm_object_pip_wakeup(dstobject);
- VM_OBJECT_WLOCK(srcobject);
- vm_object_pip_wakeup(srcobject);
- } else {
- /*
* Destination has valid swapblk or it is represented
-* by a resident page. We destroy the sourceblock.
+* by a resident page. We destroy the source block.
*/
svn commit: r337282 - head/contrib/llvm/tools/lld/ELF/Arch
Author: alc
Date: Sat Aug 4 02:30:51 2018
New Revision: 337282
URL: https://svnweb.freebsd.org/changeset/base/337282
Log:
Set the default image base on arm64 and i386 to a superpage-aligned
address.
Reviewed by: emaste, markj
Discussed with: dim
Differential Revision:https://reviews.freebsd.org/D16385
Modified:
head/contrib/llvm/tools/lld/ELF/Arch/AArch64.cpp
head/contrib/llvm/tools/lld/ELF/Arch/X86.cpp
Modified: head/contrib/llvm/tools/lld/ELF/Arch/AArch64.cpp
==
--- head/contrib/llvm/tools/lld/ELF/Arch/AArch64.cppSat Aug 4 01:45:17
2018(r337281)
+++ head/contrib/llvm/tools/lld/ELF/Arch/AArch64.cppSat Aug 4 02:30:51
2018(r337282)
@@ -66,6 +66,10 @@ AArch64::AArch64() {
PltHeaderSize = 32;
DefaultMaxPageSize = 65536;
+ // Align to the 2 MiB page size (known as a superpage or huge page).
+ // FreeBSD automatically promotes 2 MiB-aligned allocations.
+ DefaultImageBase = 0x20;
+
// It doesn't seem to be documented anywhere, but tls on aarch64 uses variant
// 1 of the tls structures and the tcb size is 16.
TcbSize = 16;
Modified: head/contrib/llvm/tools/lld/ELF/Arch/X86.cpp
==
--- head/contrib/llvm/tools/lld/ELF/Arch/X86.cppSat Aug 4 01:45:17
2018(r337281)
+++ head/contrib/llvm/tools/lld/ELF/Arch/X86.cppSat Aug 4 02:30:51
2018(r337282)
@@ -61,6 +61,10 @@ X86::X86() {
PltHeaderSize = 16;
TlsGdRelaxSkip = 2;
TrapInstr = 0x; // 0xcc = INT3
+
+ // Align to the non-PAE large page size (known as a superpage or huge page).
+ // FreeBSD automatically promotes large, superpage-aligned allocations.
+ DefaultImageBase = 0x40;
}
static bool hasBaseReg(uint8_t ModRM) { return (ModRM & 0xc7) != 0x5; }
___
svn-src-all@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-all
To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"
svn commit: r336984 - head/sys/vm
Author: alc
Date: Tue Jul 31 17:41:48 2018
New Revision: 336984
URL: https://svnweb.freebsd.org/changeset/base/336984
Log:
Allow vm object coalescing to occur in the midst of a vm object when the
OBJ_ONEMAPPING flag is set. In other words, allow recycling of existing
but unused subranges of a vm object when the OBJ_ONEMAPPING flag is set.
Such situations are increasingly common with jemalloc >= 5.0. This
change has the expected effect of reducing the number of vm map entry and
object allocations and increasing the number of superpage promotions.
Reviewed by: kib, markj
Tested by:pho
MFC after:6 weeks
Differential Revision:https://reviews.freebsd.org/D16501
Modified:
head/sys/vm/vm_object.c
Modified: head/sys/vm/vm_object.c
==
--- head/sys/vm/vm_object.c Tue Jul 31 17:18:58 2018(r336983)
+++ head/sys/vm/vm_object.c Tue Jul 31 17:41:48 2018(r336984)
@@ -2142,8 +2142,9 @@ vm_object_coalesce(vm_object_t prev_object, vm_ooffset
next_size >>= PAGE_SHIFT;
next_pindex = OFF_TO_IDX(prev_offset) + prev_size;
- if ((prev_object->ref_count > 1) &&
- (prev_object->size != next_pindex)) {
+ if (prev_object->ref_count > 1 &&
+ prev_object->size != next_pindex &&
+ (prev_object->flags & OBJ_ONEMAPPING) == 0) {
VM_OBJECT_WUNLOCK(prev_object);
return (FALSE);
}
___
svn-src-all@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-all
To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"
svn commit: r336881 - head/sys/arm/arm
Author: alc
Date: Mon Jul 30 01:54:25 2018
New Revision: 336881
URL: https://svnweb.freebsd.org/changeset/base/336881
Log:
Prepare for adding psind == 1 support to armv6's pmap_enter().
Precompute the new PTE before entering the critical section.
Eliminate duplication of the pmap and pv list unlock operations in
pmap_enter() by implementing a single return path. Otherwise, the
duplication will only increase with the upcoming support for psind == 1.
Reviewed by: mmel
Tested by:mmel
Discussed with: kib, markj
MFC after:3 weeks
Differential Revision:https://reviews.freebsd.org/D16443
Modified:
head/sys/arm/arm/pmap-v6.c
Modified: head/sys/arm/arm/pmap-v6.c
==
--- head/sys/arm/arm/pmap-v6.c Mon Jul 30 00:08:36 2018(r336880)
+++ head/sys/arm/arm/pmap-v6.c Mon Jul 30 01:54:25 2018(r336881)
@@ -3849,18 +3849,36 @@ pmap_enter(pmap_t pmap, vm_offset_t va, vm_page_t m, v
pv_entry_t pv;
vm_paddr_t opa, pa;
vm_page_t mpte2, om;
- boolean_t wired;
+ int rv;
va = trunc_page(va);
- mpte2 = NULL;
- wired = (flags & PMAP_ENTER_WIRED) != 0;
-
KASSERT(va <= vm_max_kernel_address, ("%s: toobig", __func__));
KASSERT(va < UPT2V_MIN_ADDRESS || va >= UPT2V_MAX_ADDRESS,
("%s: invalid to pmap_enter page table pages (va: 0x%x)", __func__,
va));
+ KASSERT((m->oflags & VPO_UNMANAGED) != 0 || va < kmi.clean_sva ||
+ va >= kmi.clean_eva,
+ ("%s: managed mapping within the clean submap", __func__));
if ((m->oflags & VPO_UNMANAGED) == 0 && !vm_page_xbusied(m))
VM_OBJECT_ASSERT_LOCKED(m->object);
+ KASSERT((flags & PMAP_ENTER_RESERVED) == 0,
+ ("%s: flags %u has reserved bits set", __func__, flags));
+ pa = VM_PAGE_TO_PHYS(m);
+ npte2 = PTE2(pa, PTE2_A, vm_page_pte2_attr(m));
+ if ((flags & VM_PROT_WRITE) == 0)
+ npte2 |= PTE2_NM;
+ if ((prot & VM_PROT_WRITE) == 0)
+ npte2 |= PTE2_RO;
+ KASSERT((npte2 & (PTE2_NM | PTE2_RO)) != PTE2_RO,
+ ("pmap_enter: flags includes VM_PROT_WRITE but prot doesn't"));
+ if ((prot & VM_PROT_EXECUTE) == 0)
+ npte2 |= PTE2_NX;
+ if ((flags & PMAP_ENTER_WIRED) != 0)
+ npte2 |= PTE2_W;
+ if (va < VM_MAXUSER_ADDRESS)
+ npte2 |= PTE2_U;
+ if (pmap != kernel_pmap)
+ npte2 |= PTE2_NG;
rw_wlock(_global_lock);
PMAP_LOCK(pmap);
@@ -3875,12 +3893,11 @@ pmap_enter(pmap_t pmap, vm_offset_t va, vm_page_t m, v
if (mpte2 == NULL) {
KASSERT((flags & PMAP_ENTER_NOSLEEP) != 0,
("pmap_allocpte2 failed with sleep allowed"));
- sched_unpin();
- rw_wunlock(_global_lock);
- PMAP_UNLOCK(pmap);
- return (KERN_RESOURCE_SHORTAGE);
+ rv = KERN_RESOURCE_SHORTAGE;
+ goto out;
}
- }
+ } else
+ mpte2 = NULL;
pte1p = pmap_pte1(pmap, va);
if (pte1_is_section(pte1_load(pte1p)))
panic("%s: attempted on 1MB page", __func__);
@@ -3889,7 +3906,6 @@ pmap_enter(pmap_t pmap, vm_offset_t va, vm_page_t m, v
panic("%s: invalid L1 page table entry va=%#x", __func__, va);
om = NULL;
- pa = VM_PAGE_TO_PHYS(m);
opte2 = pte2_load(pte2p);
opa = pte2_pa(opte2);
/*
@@ -3902,9 +3918,9 @@ pmap_enter(pmap_t pmap, vm_offset_t va, vm_page_t m, v
* are valid mappings in them. Hence, if a user page is wired,
* the PT2 page will be also.
*/
- if (wired && !pte2_is_wired(opte2))
+ if (pte2_is_wired(npte2) && !pte2_is_wired(opte2))
pmap->pm_stats.wired_count++;
- else if (!wired && pte2_is_wired(opte2))
+ else if (!pte2_is_wired(npte2) && pte2_is_wired(opte2))
pmap->pm_stats.wired_count--;
/*
@@ -3953,11 +3969,10 @@ pmap_enter(pmap_t pmap, vm_offset_t va, vm_page_t m, v
* Enter on the PV list if part of our managed memory.
*/
if ((m->oflags & VPO_UNMANAGED) == 0) {
- KASSERT(va < kmi.clean_sva || va >= kmi.clean_eva,
- ("%s: managed mapping within the clean submap", __func__));
- if (pv == NULL)
+ if (pv == NULL) {
pv = get_pv_entry(pmap, FALSE);
- pv->pv_va = va;
+ pv->pv_va = va;
+ }
TAILQ_INSERT_TAIL(>md.pv_list, pv, pv_next);
} else if (pv != NULL)
free_pv_entry(pmap, pv);
@@ -3965,28 +3980,17 @@
svn commit: r336803 - head/sys/vm
Author: alc
Date: Sat Jul 28 04:06:33 2018
New Revision: 336803
URL: https://svnweb.freebsd.org/changeset/base/336803
Log:
To date, mlockall(MCL_FUTURE) has had the unfortunate side effect of
blocking vm map entry and object coalescing for the calling process.
However, there is no reason that mlockall(MCL_FUTURE) should block
such coalescing. This change enables it.
Reviewed by: kib, markj
Tested by:pho
MFC after:6 weeks
Differential Revision:https://reviews.freebsd.org/D16413
Modified:
head/sys/vm/vm_map.c
Modified: head/sys/vm/vm_map.c
==
--- head/sys/vm/vm_map.cSat Jul 28 02:53:36 2018(r336802)
+++ head/sys/vm/vm_map.cSat Jul 28 04:06:33 2018(r336803)
@@ -1277,10 +1277,9 @@ charged:
vm_object_clear_flag(object, OBJ_ONEMAPPING);
VM_OBJECT_WUNLOCK(object);
} else if (prev_entry != >header &&
- prev_entry->eflags == protoeflags &&
+ (prev_entry->eflags & ~MAP_ENTRY_USER_WIRED) == protoeflags &&
(cow & (MAP_STACK_GROWS_DOWN | MAP_STACK_GROWS_UP)) == 0 &&
- prev_entry->end == start && prev_entry->wired_count == 0 &&
- (prev_entry->cred == cred ||
+ prev_entry->end == start && (prev_entry->cred == cred ||
(prev_entry->object.vm_object != NULL &&
prev_entry->object.vm_object->cred == cred)) &&
vm_object_coalesce(prev_entry->object.vm_object,
@@ -1295,7 +1294,11 @@ charged:
*/
if (prev_entry->inheritance == inheritance &&
prev_entry->protection == prot &&
- prev_entry->max_protection == max) {
+ prev_entry->max_protection == max &&
+ prev_entry->wired_count == 0) {
+ KASSERT((prev_entry->eflags & MAP_ENTRY_USER_WIRED) ==
+ 0, ("prev_entry %p has incoherent wiring",
+ prev_entry));
if ((prev_entry->eflags & MAP_ENTRY_GUARD) == 0)
map->size += end - prev_entry->end;
prev_entry->end = end;
___
svn-src-all@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-all
To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"
svn commit: r336589 - head/sys/arm64/arm64
Author: alc Date: Sat Jul 21 21:26:38 2018 New Revision: 336589 URL: https://svnweb.freebsd.org/changeset/base/336589 Log: Eliminate a comment that doesn't apply to this pmap implementation. Coalesce the variable definitions for PV entry management. MFC after:3 weeks Modified: head/sys/arm64/arm64/pmap.c Modified: head/sys/arm64/arm64/pmap.c == --- head/sys/arm64/arm64/pmap.c Sat Jul 21 20:14:01 2018(r336588) +++ head/sys/arm64/arm64/pmap.c Sat Jul 21 21:26:38 2018(r336589) @@ -242,9 +242,10 @@ vm_offset_t kernel_vm_end = 0; /* * Data for the pv entry allocation mechanism. - * Updates to pv_invl_gen are protected by the pv_list_locks[] - * elements, but reads are not. */ +static TAILQ_HEAD(pch, pv_chunk) pv_chunks = TAILQ_HEAD_INITIALIZER(pv_chunks); +static struct mtx pv_chunks_mutex; +static struct rwlock pv_list_locks[NPV_LIST_LOCKS]; static struct md_page *pv_table; static struct md_page pv_dummy; @@ -269,13 +270,6 @@ static int superpages_enabled = 1; SYSCTL_INT(_vm_pmap, OID_AUTO, superpages_enabled, CTLFLAG_RDTUN | CTLFLAG_NOFETCH, _enabled, 0, "Are large page mappings enabled?"); - -/* - * Data for the pv entry allocation mechanism - */ -static TAILQ_HEAD(pch, pv_chunk) pv_chunks = TAILQ_HEAD_INITIALIZER(pv_chunks); -static struct mtx pv_chunks_mutex; -static struct rwlock pv_list_locks[NPV_LIST_LOCKS]; /* * Internal flags for pmap_enter()'s helper functions. ___ svn-src-all@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"
