[PATCH v5 09/32] x86/mm: Provide general kernel support for memory encryption
Changes to the existing page table macros will allow the SME support to
be enabled in a simple fashion with minimal changes to files that use these
macros. Since the memory encryption mask will now be part of the regular
pagetable macros, we introduce two new macros (_PAGE_TABLE_NOENC and
_KERNPG_TABLE_NOENC) to allow for early pagetable creation/initialization
without the encryption mask before SME becomes active. Two new pgprot()
macros are defined to allow setting or clearing the page encryption mask.
The FIXMAP_PAGE_NOCACHE define is introduced for use with MMIO. SME does
not support encryption for MMIO areas so this define removes the encryption
mask from the page attribute.
Two new macros are introduced (__sme_pa() / __sme_pa_nodebug()) to allow
creating a physical address with the encryption mask. These are used when
working with the cr3 register so that the PGD can be encrypted. The current
__va() macro is updated so that the virtual address is generated based off
of the physical address without the encryption mask thus allowing the same
virtual address to be generated regardless of whether encryption is enabled
for that physical location or not.
Also, an early initialization function is added for SME. If SME is active,
this function:
- Updates the early_pmd_flags so that early page faults create mappings
with the encryption mask.
- Updates the __supported_pte_mask to include the encryption mask.
- Updates the protection_map entries to include the encryption mask so
that user-space allocations will automatically have the encryption mask
applied.
Signed-off-by: Tom Lendacky
---
arch/x86/boot/compressed/pagetable.c |7 +
arch/x86/include/asm/fixmap.h|7 +
arch/x86/include/asm/mem_encrypt.h | 25 +++
arch/x86/include/asm/page.h |4 ++-
arch/x86/include/asm/page_types.h|2 +-
arch/x86/include/asm/pgtable.h |9 +++
arch/x86/include/asm/pgtable_types.h | 45 ++
arch/x86/include/asm/processor.h |3 ++
arch/x86/kernel/espfix_64.c |2 +-
arch/x86/kernel/head64.c | 12 -
arch/x86/kernel/head_64.S| 18 +++---
arch/x86/mm/kasan_init_64.c |4 ++-
arch/x86/mm/mem_encrypt.c| 18 ++
arch/x86/mm/pageattr.c |3 ++
include/asm-generic/pgtable.h|8 ++
15 files changed, 134 insertions(+), 33 deletions(-)
diff --git a/arch/x86/boot/compressed/pagetable.c
b/arch/x86/boot/compressed/pagetable.c
index 56589d0..411c443 100644
--- a/arch/x86/boot/compressed/pagetable.c
+++ b/arch/x86/boot/compressed/pagetable.c
@@ -15,6 +15,13 @@
#define __pa(x) ((unsigned long)(x))
#define __va(x) ((void *)((unsigned long)(x)))
+/*
+ * The pgtable.h and mm/ident_map.c includes make use of the SME related
+ * information which is not used in the compressed image support. Un-define
+ * the SME support to avoid any compile and link errors.
+ */
+#undef CONFIG_AMD_MEM_ENCRYPT
+
#include "misc.h"
/* These actually do the work of building the kernel identity maps. */
diff --git a/arch/x86/include/asm/fixmap.h b/arch/x86/include/asm/fixmap.h
index b65155c..d9ff226 100644
--- a/arch/x86/include/asm/fixmap.h
+++ b/arch/x86/include/asm/fixmap.h
@@ -157,6 +157,13 @@ static inline void __set_fixmap(enum fixed_addresses idx,
}
#endif
+/*
+ * FIXMAP_PAGE_NOCACHE is used for MMIO. Memory encryption is not
+ * supported for MMIO addresses, so make sure that the memory encryption
+ * mask is not part of the page attributes.
+ */
+#define FIXMAP_PAGE_NOCACHE PAGE_KERNEL_IO_NOCACHE
+
#include
#define __late_set_fixmap(idx, phys, flags) __set_fixmap(idx, phys, flags)
diff --git a/arch/x86/include/asm/mem_encrypt.h
b/arch/x86/include/asm/mem_encrypt.h
index d5c4a2b..9fdbc53 100644
--- a/arch/x86/include/asm/mem_encrypt.h
+++ b/arch/x86/include/asm/mem_encrypt.h
@@ -15,6 +15,8 @@
#ifndef __ASSEMBLY__
+#include
+
#ifdef CONFIG_AMD_MEM_ENCRYPT
extern unsigned long sme_me_mask;
@@ -24,6 +26,8 @@ static inline bool sme_active(void)
return !!sme_me_mask;
}
+void __init sme_early_init(void);
+
#else /* !CONFIG_AMD_MEM_ENCRYPT */
#ifndef sme_me_mask
@@ -35,8 +39,29 @@ static inline bool sme_active(void)
}
#endif
+static inline void __init sme_early_init(void)
+{
+}
+
#endif /* CONFIG_AMD_MEM_ENCRYPT */
+/*
+ * The __sme_pa() and __sme_pa_nodebug() macros are meant for use when
+ * writing to or comparing values from the cr3 register. Having the
+ * encryption mask set in cr3 enables the PGD entry to be encrypted and
+ * avoid special case handling of PGD allocations.
+ */
+#define __sme_pa(x)(__pa(x) | sme_me_mask)
+#define __sme_pa_nodebug(x)(__pa_nodebug(x) | sme_me_mask)
+
+/*
+ * The __sme_set() and __sme_clr() macros are useful for adding or removing
+ * the encryption mask from a value (e.g. when dealing with pagetable
+ *
Re: [PATCH v5 09/32] x86/mm: Provide general kernel support for memory encryption
On 04/18/2017 02:17 PM, Tom Lendacky wrote: > @@ -55,7 +57,7 @@ static inline void copy_user_page(void *to, void *from, > unsigned long vaddr, > __phys_addr_symbol(__phys_reloc_hide((unsigned long)(x))) > > #ifndef __va > -#define __va(x) ((void *)((unsigned > long)(x)+PAGE_OFFSET)) > +#define __va(x) ((void *)(__sme_clr(x) + PAGE_OFFSET)) > #endif It seems wrong to be modifying __va(). It currently takes a physical address, and this modifies it to take a physical address plus the SME bits. How does that end up ever happening? If we are pulling physical addresses out of the page tables, we use p??_phys(). I'd expect *those* to be masking off the SME bits. Is it these cases? pgd_t *base = __va(read_cr3()); For those, it seems like we really want to create two modes of reading cr3. One that truly reads CR3 and another that reads the pgd's physical address out of CR3. Then you only do the SME masking on the one fetching a physical address, and the SME bits never leak into __va(). ___ kexec mailing list kexec@lists.infradead.org http://lists.infradead.org/mailman/listinfo/kexec
Re: [PATCH v5 09/32] x86/mm: Provide general kernel support for memory encryption
On 4/21/2017 4:52 PM, Dave Hansen wrote: On 04/18/2017 02:17 PM, Tom Lendacky wrote: @@ -55,7 +57,7 @@ static inline void copy_user_page(void *to, void *from, unsigned long vaddr, __phys_addr_symbol(__phys_reloc_hide((unsigned long)(x))) #ifndef __va -#define __va(x)((void *)((unsigned long)(x)+PAGE_OFFSET)) +#define __va(x)((void *)(__sme_clr(x) + PAGE_OFFSET)) #endif It seems wrong to be modifying __va(). It currently takes a physical address, and this modifies it to take a physical address plus the SME bits. This actually modifies it to be sure the encryption bit is not part of the physical address. How does that end up ever happening? If we are pulling physical addresses out of the page tables, we use p??_phys(). I'd expect *those* to be masking off the SME bits. Is it these cases? pgd_t *base = __va(read_cr3()); For those, it seems like we really want to create two modes of reading cr3. One that truly reads CR3 and another that reads the pgd's physical address out of CR3. Then you only do the SME masking on the one fetching a physical address, and the SME bits never leak into __va(). I'll investigate this and see if I can remove the mod to __va(). Thanks, Tom ___ kexec mailing list kexec@lists.infradead.org http://lists.infradead.org/mailman/listinfo/kexec
Re: [PATCH v5 09/32] x86/mm: Provide general kernel support for memory encryption
On 04/24/2017 08:53 AM, Tom Lendacky wrote: > On 4/21/2017 4:52 PM, Dave Hansen wrote: >> On 04/18/2017 02:17 PM, Tom Lendacky wrote: >>> @@ -55,7 +57,7 @@ static inline void copy_user_page(void *to, void >>> *from, unsigned long vaddr, >>> __phys_addr_symbol(__phys_reloc_hide((unsigned long)(x))) >>> >>> #ifndef __va >>> -#define __va(x)((void *)((unsigned long)(x)+PAGE_OFFSET)) >>> +#define __va(x)((void *)(__sme_clr(x) + PAGE_OFFSET)) >>> #endif >> >> It seems wrong to be modifying __va(). It currently takes a physical >> address, and this modifies it to take a physical address plus the SME >> bits. > > This actually modifies it to be sure the encryption bit is not part of > the physical address. If SME bits make it this far, we have a bug elsewhere. Right? Probably best not to paper over it. ___ kexec mailing list kexec@lists.infradead.org http://lists.infradead.org/mailman/listinfo/kexec
Re: [PATCH v5 09/32] x86/mm: Provide general kernel support for memory encryption
On 4/24/2017 10:57 AM, Dave Hansen wrote: On 04/24/2017 08:53 AM, Tom Lendacky wrote: On 4/21/2017 4:52 PM, Dave Hansen wrote: On 04/18/2017 02:17 PM, Tom Lendacky wrote: @@ -55,7 +57,7 @@ static inline void copy_user_page(void *to, void *from, unsigned long vaddr, __phys_addr_symbol(__phys_reloc_hide((unsigned long)(x))) #ifndef __va -#define __va(x)((void *)((unsigned long)(x)+PAGE_OFFSET)) +#define __va(x)((void *)(__sme_clr(x) + PAGE_OFFSET)) #endif It seems wrong to be modifying __va(). It currently takes a physical address, and this modifies it to take a physical address plus the SME bits. This actually modifies it to be sure the encryption bit is not part of the physical address. If SME bits make it this far, we have a bug elsewhere. Right? Probably best not to paper over it. That all depends on the approach. Currently that's not the case for the one situation that you mentioned with cr3. But if we do take the approach that we should never feed physical addresses to __va() with the encryption bit set then, yes, it would imply a bug elsewhere - which is probably a good approach. I'll work on that. I could even add a debug config option that would issue a warning should __va() encounter the encryption bit if SME is enabled or active. Thanks, Tom ___ kexec mailing list kexec@lists.infradead.org http://lists.infradead.org/mailman/listinfo/kexec
Re: [PATCH v5 09/32] x86/mm: Provide general kernel support for memory encryption
On Tue, Apr 18, 2017 at 04:17:54PM -0500, Tom Lendacky wrote: > Changes to the existing page table macros will allow the SME support to > be enabled in a simple fashion with minimal changes to files that use these > macros. Since the memory encryption mask will now be part of the regular > pagetable macros, we introduce two new macros (_PAGE_TABLE_NOENC and > _KERNPG_TABLE_NOENC) to allow for early pagetable creation/initialization > without the encryption mask before SME becomes active. Two new pgprot() > macros are defined to allow setting or clearing the page encryption mask. ... > @@ -55,7 +57,7 @@ static inline void copy_user_page(void *to, void *from, > unsigned long vaddr, > __phys_addr_symbol(__phys_reloc_hide((unsigned long)(x))) > > #ifndef __va > -#define __va(x) ((void *)((unsigned > long)(x)+PAGE_OFFSET)) > +#define __va(x) ((void *)(__sme_clr(x) + PAGE_OFFSET)) > #endif > > #define __boot_va(x) __va(x) > diff --git a/arch/x86/include/asm/page_types.h > b/arch/x86/include/asm/page_types.h > index 7bd0099..fead0a5 100644 > --- a/arch/x86/include/asm/page_types.h > +++ b/arch/x86/include/asm/page_types.h > @@ -15,7 +15,7 @@ > #define PUD_PAGE_SIZE(_AC(1, UL) << PUD_SHIFT) > #define PUD_PAGE_MASK(~(PUD_PAGE_SIZE-1)) > > -#define __PHYSICAL_MASK ((phys_addr_t)((1ULL << > __PHYSICAL_MASK_SHIFT) - 1)) > +#define __PHYSICAL_MASK ((phys_addr_t)(__sme_clr((1ULL << > __PHYSICAL_MASK_SHIFT) - 1))) That looks strange: poking SME mask hole into a mask...? > #define __VIRTUAL_MASK ((1UL << __VIRTUAL_MASK_SHIFT) - 1) > > /* Cast *PAGE_MASK to a signed type so that it is sign-extended if -- Regards/Gruss, Boris. Good mailing practices for 400: avoid top-posting and trim the reply. ___ kexec mailing list kexec@lists.infradead.org http://lists.infradead.org/mailman/listinfo/kexec
Re: [PATCH v5 09/32] x86/mm: Provide general kernel support for memory encryption
On 4/27/2017 11:12 AM, Borislav Petkov wrote: On Tue, Apr 18, 2017 at 04:17:54PM -0500, Tom Lendacky wrote: Changes to the existing page table macros will allow the SME support to be enabled in a simple fashion with minimal changes to files that use these macros. Since the memory encryption mask will now be part of the regular pagetable macros, we introduce two new macros (_PAGE_TABLE_NOENC and _KERNPG_TABLE_NOENC) to allow for early pagetable creation/initialization without the encryption mask before SME becomes active. Two new pgprot() macros are defined to allow setting or clearing the page encryption mask. ... @@ -55,7 +57,7 @@ static inline void copy_user_page(void *to, void *from, unsigned long vaddr, __phys_addr_symbol(__phys_reloc_hide((unsigned long)(x))) #ifndef __va -#define __va(x)((void *)((unsigned long)(x)+PAGE_OFFSET)) +#define __va(x)((void *)(__sme_clr(x) + PAGE_OFFSET)) #endif #define __boot_va(x) __va(x) diff --git a/arch/x86/include/asm/page_types.h b/arch/x86/include/asm/page_types.h index 7bd0099..fead0a5 100644 --- a/arch/x86/include/asm/page_types.h +++ b/arch/x86/include/asm/page_types.h @@ -15,7 +15,7 @@ #define PUD_PAGE_SIZE (_AC(1, UL) << PUD_SHIFT) #define PUD_PAGE_MASK (~(PUD_PAGE_SIZE-1)) -#define __PHYSICAL_MASK((phys_addr_t)((1ULL << __PHYSICAL_MASK_SHIFT) - 1)) +#define __PHYSICAL_MASK((phys_addr_t)(__sme_clr((1ULL << __PHYSICAL_MASK_SHIFT) - 1))) That looks strange: poking SME mask hole into a mask...? I masked it out here based on a previous comment from Dave Hansen: http://marc.info/?l=linux-kernel&m=148778719826905&w=2 I could move the __sme_clr into the individual defines of: PHYSICAL_PAGE_MASK, PHYSICAL_PMD_PAGE_MASK and PHYSICAL_PUD_PAGE_MASK Either way works for me. Thanks, Tom #define __VIRTUAL_MASK ((1UL << __VIRTUAL_MASK_SHIFT) - 1) /* Cast *PAGE_MASK to a signed type so that it is sign-extended if ___ kexec mailing list kexec@lists.infradead.org http://lists.infradead.org/mailman/listinfo/kexec
Re: [PATCH v5 09/32] x86/mm: Provide general kernel support for memory encryption
On Thu, May 04, 2017 at 09:34:09AM -0500, Tom Lendacky wrote: > I masked it out here based on a previous comment from Dave Hansen: > > http://marc.info/?l=linux-kernel&m=148778719826905&w=2 > > I could move the __sme_clr into the individual defines of: Nah, it is fine as it is. I was just wondering... Thanks. -- Regards/Gruss, Boris. Good mailing practices for 400: avoid top-posting and trim the reply. ___ kexec mailing list kexec@lists.infradead.org http://lists.infradead.org/mailman/listinfo/kexec
