Re: [PATCH 21/30] x86, mm: put mmu-to-h/w ASID translation in one place

[Andy Lutomirski]

On Fri, Nov 10, 2017 at 2:09 PM, Dave Hansen
 wrote:
> On 11/10/2017 02:03 PM, Andy Lutomirski wrote:
>>> +static inline u16 kern_asid(u16 asid)
>>> +{
>>> +   VM_WARN_ON_ONCE(asid > MAX_ASID_AVAILABLE);
>>> +   /*
>>> +* If PCID is on, ASID-aware code paths put the ASID+1 into the PCID
>>> +* bits.  This serves two purposes.  It prevents a nasty situation 
>>> in
>>> +* which PCID-unaware code saves CR3, loads some other value (with 
>>> PCID
>>> +* == 0), and then restores CR3, thus corrupting the TLB for ASID 0 
>>> if
>>> +* the saved ASID was nonzero.  It also means that any bugs 
>>> involving
>>> +* loading a PCID-enabled CR3 with CR4.PCIDE off will trigger
>>> +* deterministically.
>>> +*/
>>> +   return asid + 1;
>>> +}
>> This seems really error-prone.  Maybe we should have a pcid_t type and
>> make all the interfaces that want a h/w PCID take pcid_t.
>
> Yeah, totally agree.  I actually had a nasty bug or two around this area
> because of this.
>
> I divided them among hw_asid_t and sw_asid_t.  You can turn a sw_asid_t
> into a kernel hw_asid_t or a user hw_asid_t.  But, it cause too much
> churn across the TLB flushing code so I shelved it for now.
>
> I'd love to come back nd fix this up properly though.

In the long run, I would go with int for the sw asid and pcid_t for
the PCID.  After all, we index arrays with the SW one.

Re: [PATCH 21/30] x86, mm: put mmu-to-h/w ASID translation in one place

[Andy Lutomirski]

On Fri, Nov 10, 2017 at 2:09 PM, Dave Hansen
 wrote:
> On 11/10/2017 02:03 PM, Andy Lutomirski wrote:
>>> +static inline u16 kern_asid(u16 asid)
>>> +{
>>> +   VM_WARN_ON_ONCE(asid > MAX_ASID_AVAILABLE);
>>> +   /*
>>> +* If PCID is on, ASID-aware code paths put the ASID+1 into the PCID
>>> +* bits.  This serves two purposes.  It prevents a nasty situation 
>>> in
>>> +* which PCID-unaware code saves CR3, loads some other value (with 
>>> PCID
>>> +* == 0), and then restores CR3, thus corrupting the TLB for ASID 0 
>>> if
>>> +* the saved ASID was nonzero.  It also means that any bugs 
>>> involving
>>> +* loading a PCID-enabled CR3 with CR4.PCIDE off will trigger
>>> +* deterministically.
>>> +*/
>>> +   return asid + 1;
>>> +}
>> This seems really error-prone.  Maybe we should have a pcid_t type and
>> make all the interfaces that want a h/w PCID take pcid_t.
>
> Yeah, totally agree.  I actually had a nasty bug or two around this area
> because of this.
>
> I divided them among hw_asid_t and sw_asid_t.  You can turn a sw_asid_t
> into a kernel hw_asid_t or a user hw_asid_t.  But, it cause too much
> churn across the TLB flushing code so I shelved it for now.
>
> I'd love to come back nd fix this up properly though.

In the long run, I would go with int for the sw asid and pcid_t for
the PCID.  After all, we index arrays with the SW one.

Re: [PATCH 21/30] x86, mm: put mmu-to-h/w ASID translation in one place

[Dave Hansen]

On 11/10/2017 02:03 PM, Andy Lutomirski wrote:
>> +static inline u16 kern_asid(u16 asid)
>> +{
>> +   VM_WARN_ON_ONCE(asid > MAX_ASID_AVAILABLE);
>> +   /*
>> +* If PCID is on, ASID-aware code paths put the ASID+1 into the PCID
>> +* bits.  This serves two purposes.  It prevents a nasty situation in
>> +* which PCID-unaware code saves CR3, loads some other value (with 
>> PCID
>> +* == 0), and then restores CR3, thus corrupting the TLB for ASID 0 
>> if
>> +* the saved ASID was nonzero.  It also means that any bugs involving
>> +* loading a PCID-enabled CR3 with CR4.PCIDE off will trigger
>> +* deterministically.
>> +*/
>> +   return asid + 1;
>> +}
> This seems really error-prone.  Maybe we should have a pcid_t type and
> make all the interfaces that want a h/w PCID take pcid_t.

Yeah, totally agree.  I actually had a nasty bug or two around this area
because of this.

I divided them among hw_asid_t and sw_asid_t.  You can turn a sw_asid_t
into a kernel hw_asid_t or a user hw_asid_t.  But, it cause too much
churn across the TLB flushing code so I shelved it for now.

I'd love to come back nd fix this up properly though.

Re: [PATCH 21/30] x86, mm: put mmu-to-h/w ASID translation in one place

[Dave Hansen]

On 11/10/2017 02:03 PM, Andy Lutomirski wrote:
>> +static inline u16 kern_asid(u16 asid)
>> +{
>> +   VM_WARN_ON_ONCE(asid > MAX_ASID_AVAILABLE);
>> +   /*
>> +* If PCID is on, ASID-aware code paths put the ASID+1 into the PCID
>> +* bits.  This serves two purposes.  It prevents a nasty situation in
>> +* which PCID-unaware code saves CR3, loads some other value (with 
>> PCID
>> +* == 0), and then restores CR3, thus corrupting the TLB for ASID 0 
>> if
>> +* the saved ASID was nonzero.  It also means that any bugs involving
>> +* loading a PCID-enabled CR3 with CR4.PCIDE off will trigger
>> +* deterministically.
>> +*/
>> +   return asid + 1;
>> +}
> This seems really error-prone.  Maybe we should have a pcid_t type and
> make all the interfaces that want a h/w PCID take pcid_t.

Yeah, totally agree.  I actually had a nasty bug or two around this area
because of this.

I divided them among hw_asid_t and sw_asid_t.  You can turn a sw_asid_t
into a kernel hw_asid_t or a user hw_asid_t.  But, it cause too much
churn across the TLB flushing code so I shelved it for now.

I'd love to come back nd fix this up properly though.

Re: [PATCH 21/30] x86, mm: put mmu-to-h/w ASID translation in one place

[Andy Lutomirski]

On Fri, Nov 10, 2017 at 11:31 AM, Dave Hansen
 wrote:
>
> From: Dave Hansen 
>
> There are effectively two ASID types:
> 1. The one stored in the mmu_context that goes from 0->5
> 2. The one programmed into the hardware that goes from 1->6
>
> This consolidates the locations where converting beween the two
> (by doing +1) to a single place which gives us a nice place to
> comment.  KAISER will also need to, given an ASID, know which
> hardware ASID to flush for the userspace mapping.
>
> Signed-off-by: Dave Hansen 
> Cc: Moritz Lipp 
> Cc: Daniel Gruss 
> Cc: Michael Schwarz 
> Cc: Richard Fellner 
> Cc: Andy Lutomirski 
> Cc: Linus Torvalds 
> Cc: Kees Cook 
> Cc: Hugh Dickins 
> Cc: x...@kernel.org
> ---
>
>  b/arch/x86/include/asm/tlbflush.h |   30 ++
>  1 file changed, 18 insertions(+), 12 deletions(-)
>
> diff -puN arch/x86/include/asm/tlbflush.h~kaiser-pcid-pre-build-kern 
> arch/x86/include/asm/tlbflush.h
> --- a/arch/x86/include/asm/tlbflush.h~kaiser-pcid-pre-build-kern
> 2017-11-10 11:22:16.521244931 -0800
> +++ b/arch/x86/include/asm/tlbflush.h   2017-11-10 11:22:16.525244931 -0800
> @@ -87,21 +87,26 @@ static inline u64 inc_mm_tlb_gen(struct
>   */
>  #define MAX_ASID_AVAILABLE ((1<
> -/*
> - * If PCID is on, ASID-aware code paths put the ASID+1 into the PCID
> - * bits.  This serves two purposes.  It prevents a nasty situation in
> - * which PCID-unaware code saves CR3, loads some other value (with PCID
> - * == 0), and then restores CR3, thus corrupting the TLB for ASID 0 if
> - * the saved ASID was nonzero.  It also means that any bugs involving
> - * loading a PCID-enabled CR3 with CR4.PCIDE off will trigger
> - * deterministically.
> - */
> +static inline u16 kern_asid(u16 asid)
> +{
> +   VM_WARN_ON_ONCE(asid > MAX_ASID_AVAILABLE);
> +   /*
> +* If PCID is on, ASID-aware code paths put the ASID+1 into the PCID
> +* bits.  This serves two purposes.  It prevents a nasty situation in
> +* which PCID-unaware code saves CR3, loads some other value (with 
> PCID
> +* == 0), and then restores CR3, thus corrupting the TLB for ASID 0 if
> +* the saved ASID was nonzero.  It also means that any bugs involving
> +* loading a PCID-enabled CR3 with CR4.PCIDE off will trigger
> +* deterministically.
> +*/
> +   return asid + 1;
> +}

This seems really error-prone.  Maybe we should have a pcid_t type and
make all the interfaces that want a h/w PCID take pcid_t.

--Andy

Re: [PATCH 21/30] x86, mm: put mmu-to-h/w ASID translation in one place

[Andy Lutomirski]

On Fri, Nov 10, 2017 at 11:31 AM, Dave Hansen
 wrote:
>
> From: Dave Hansen 
>
> There are effectively two ASID types:
> 1. The one stored in the mmu_context that goes from 0->5
> 2. The one programmed into the hardware that goes from 1->6
>
> This consolidates the locations where converting beween the two
> (by doing +1) to a single place which gives us a nice place to
> comment.  KAISER will also need to, given an ASID, know which
> hardware ASID to flush for the userspace mapping.
>
> Signed-off-by: Dave Hansen 
> Cc: Moritz Lipp 
> Cc: Daniel Gruss 
> Cc: Michael Schwarz 
> Cc: Richard Fellner 
> Cc: Andy Lutomirski 
> Cc: Linus Torvalds 
> Cc: Kees Cook 
> Cc: Hugh Dickins 
> Cc: x...@kernel.org
> ---
>
>  b/arch/x86/include/asm/tlbflush.h |   30 ++
>  1 file changed, 18 insertions(+), 12 deletions(-)
>
> diff -puN arch/x86/include/asm/tlbflush.h~kaiser-pcid-pre-build-kern 
> arch/x86/include/asm/tlbflush.h
> --- a/arch/x86/include/asm/tlbflush.h~kaiser-pcid-pre-build-kern
> 2017-11-10 11:22:16.521244931 -0800
> +++ b/arch/x86/include/asm/tlbflush.h   2017-11-10 11:22:16.525244931 -0800
> @@ -87,21 +87,26 @@ static inline u64 inc_mm_tlb_gen(struct
>   */
>  #define MAX_ASID_AVAILABLE ((1<
> -/*
> - * If PCID is on, ASID-aware code paths put the ASID+1 into the PCID
> - * bits.  This serves two purposes.  It prevents a nasty situation in
> - * which PCID-unaware code saves CR3, loads some other value (with PCID
> - * == 0), and then restores CR3, thus corrupting the TLB for ASID 0 if
> - * the saved ASID was nonzero.  It also means that any bugs involving
> - * loading a PCID-enabled CR3 with CR4.PCIDE off will trigger
> - * deterministically.
> - */
> +static inline u16 kern_asid(u16 asid)
> +{
> +   VM_WARN_ON_ONCE(asid > MAX_ASID_AVAILABLE);
> +   /*
> +* If PCID is on, ASID-aware code paths put the ASID+1 into the PCID
> +* bits.  This serves two purposes.  It prevents a nasty situation in
> +* which PCID-unaware code saves CR3, loads some other value (with 
> PCID
> +* == 0), and then restores CR3, thus corrupting the TLB for ASID 0 if
> +* the saved ASID was nonzero.  It also means that any bugs involving
> +* loading a PCID-enabled CR3 with CR4.PCIDE off will trigger
> +* deterministically.
> +*/
> +   return asid + 1;
> +}

This seems really error-prone.  Maybe we should have a pcid_t type and
make all the interfaces that want a h/w PCID take pcid_t.

--Andy

[PATCH 21/30] x86, mm: put mmu-to-h/w ASID translation in one place

[Dave Hansen]

From: Dave Hansen 

There are effectively two ASID types:
1. The one stored in the mmu_context that goes from 0->5
2. The one programmed into the hardware that goes from 1->6

This consolidates the locations where converting beween the two
(by doing +1) to a single place which gives us a nice place to
comment.  KAISER will also need to, given an ASID, know which
hardware ASID to flush for the userspace mapping.

Signed-off-by: Dave Hansen 
Cc: Moritz Lipp 
Cc: Daniel Gruss 
Cc: Michael Schwarz 
Cc: Richard Fellner 
Cc: Andy Lutomirski 
Cc: Linus Torvalds 
Cc: Kees Cook 
Cc: Hugh Dickins 
Cc: x...@kernel.org
---

 b/arch/x86/include/asm/tlbflush.h |   30 ++
 1 file changed, 18 insertions(+), 12 deletions(-)

diff -puN arch/x86/include/asm/tlbflush.h~kaiser-pcid-pre-build-kern 
arch/x86/include/asm/tlbflush.h
--- a/arch/x86/include/asm/tlbflush.h~kaiser-pcid-pre-build-kern
2017-11-10 11:22:16.521244931 -0800
+++ b/arch/x86/include/asm/tlbflush.h   2017-11-10 11:22:16.525244931 -0800
@@ -87,21 +87,26 @@ static inline u64 inc_mm_tlb_gen(struct
  */
 #define MAX_ASID_AVAILABLE ((1< MAX_ASID_AVAILABLE);
+   /*
+* If PCID is on, ASID-aware code paths put the ASID+1 into the PCID
+* bits.  This serves two purposes.  It prevents a nasty situation in
+* which PCID-unaware code saves CR3, loads some other value (with PCID
+* == 0), and then restores CR3, thus corrupting the TLB for ASID 0 if
+* the saved ASID was nonzero.  It also means that any bugs involving
+* loading a PCID-enabled CR3 with CR4.PCIDE off will trigger
+* deterministically.
+*/
+   return asid + 1;
+}
+
 struct pgd_t;
 static inline unsigned long build_cr3(pgd_t *pgd, u16 asid)
 {
if (static_cpu_has(X86_FEATURE_PCID)) {
-   VM_WARN_ON_ONCE(asid > MAX_ASID_AVAILABLE);
-   return __sme_pa(pgd) | (asid + 1);
+   return __sme_pa(pgd) | kern_asid(asid);
} else {
VM_WARN_ON_ONCE(asid != 0);
return __sme_pa(pgd);
@@ -111,7 +116,8 @@ static inline unsigned long build_cr3(pg
 static inline unsigned long build_cr3_noflush(pgd_t *pgd, u16 asid)
 {
VM_WARN_ON_ONCE(asid > MAX_ASID_AVAILABLE);
-   return __sme_pa(pgd) | (asid + 1) | CR3_NOFLUSH;
+   VM_WARN_ON_ONCE(!this_cpu_has(X86_FEATURE_PCID));
+   return __sme_pa(pgd) | kern_asid(asid) | CR3_NOFLUSH;
 }
 
 #ifdef CONFIG_PARAVIRT
_

[PATCH 21/30] x86, mm: put mmu-to-h/w ASID translation in one place

[Dave Hansen]

From: Dave Hansen 

There are effectively two ASID types:
1. The one stored in the mmu_context that goes from 0->5
2. The one programmed into the hardware that goes from 1->6

This consolidates the locations where converting beween the two
(by doing +1) to a single place which gives us a nice place to
comment.  KAISER will also need to, given an ASID, know which
hardware ASID to flush for the userspace mapping.

Signed-off-by: Dave Hansen 
Cc: Moritz Lipp 
Cc: Daniel Gruss 
Cc: Michael Schwarz 
Cc: Richard Fellner 
Cc: Andy Lutomirski 
Cc: Linus Torvalds 
Cc: Kees Cook 
Cc: Hugh Dickins 
Cc: x...@kernel.org
---

 b/arch/x86/include/asm/tlbflush.h |   30 ++
 1 file changed, 18 insertions(+), 12 deletions(-)

diff -puN arch/x86/include/asm/tlbflush.h~kaiser-pcid-pre-build-kern 
arch/x86/include/asm/tlbflush.h
--- a/arch/x86/include/asm/tlbflush.h~kaiser-pcid-pre-build-kern
2017-11-10 11:22:16.521244931 -0800
+++ b/arch/x86/include/asm/tlbflush.h   2017-11-10 11:22:16.525244931 -0800
@@ -87,21 +87,26 @@ static inline u64 inc_mm_tlb_gen(struct
  */
 #define MAX_ASID_AVAILABLE ((1< MAX_ASID_AVAILABLE);
+   /*
+* If PCID is on, ASID-aware code paths put the ASID+1 into the PCID
+* bits.  This serves two purposes.  It prevents a nasty situation in
+* which PCID-unaware code saves CR3, loads some other value (with PCID
+* == 0), and then restores CR3, thus corrupting the TLB for ASID 0 if
+* the saved ASID was nonzero.  It also means that any bugs involving
+* loading a PCID-enabled CR3 with CR4.PCIDE off will trigger
+* deterministically.
+*/
+   return asid + 1;
+}
+
 struct pgd_t;
 static inline unsigned long build_cr3(pgd_t *pgd, u16 asid)
 {
if (static_cpu_has(X86_FEATURE_PCID)) {
-   VM_WARN_ON_ONCE(asid > MAX_ASID_AVAILABLE);
-   return __sme_pa(pgd) | (asid + 1);
+   return __sme_pa(pgd) | kern_asid(asid);
} else {
VM_WARN_ON_ONCE(asid != 0);
return __sme_pa(pgd);
@@ -111,7 +116,8 @@ static inline unsigned long build_cr3(pg
 static inline unsigned long build_cr3_noflush(pgd_t *pgd, u16 asid)
 {
VM_WARN_ON_ONCE(asid > MAX_ASID_AVAILABLE);
-   return __sme_pa(pgd) | (asid + 1) | CR3_NOFLUSH;
+   VM_WARN_ON_ONCE(!this_cpu_has(X86_FEATURE_PCID));
+   return __sme_pa(pgd) | kern_asid(asid) | CR3_NOFLUSH;
 }
 
 #ifdef CONFIG_PARAVIRT
_

[PATCH 21/30] x86, mm: put mmu-to-h/w ASID translation in one place

[Dave Hansen]

From: Dave Hansen 

We effectively have two ASID types:
1. The one stored in the mmu_context that goes from 0->5
2. The one we program into the hardware that goes from 1->6

Let's just put the +1 in a single place which gives us a
nice place to comment.  KAISER will also need to, given an
ASID, know which hardware ASID to flush for the userspace
mapping.

Signed-off-by: Dave Hansen 
Cc: Moritz Lipp 
Cc: Daniel Gruss 
Cc: Michael Schwarz 
Cc: Richard Fellner 
Cc: Andy Lutomirski 
Cc: Linus Torvalds 
Cc: Kees Cook 
Cc: Hugh Dickins 
Cc: x...@kernel.org
---

 b/arch/x86/include/asm/tlbflush.h |   30 ++
 1 file changed, 18 insertions(+), 12 deletions(-)

diff -puN arch/x86/include/asm/tlbflush.h~kaiser-pcid-pre-build-kern 
arch/x86/include/asm/tlbflush.h
--- a/arch/x86/include/asm/tlbflush.h~kaiser-pcid-pre-build-kern
2017-11-08 10:45:37.314681375 -0800
+++ b/arch/x86/include/asm/tlbflush.h   2017-11-08 10:45:37.317681375 -0800
@@ -86,21 +86,26 @@ static inline u64 inc_mm_tlb_gen(struct
  */
 #define NR_AVAIL_ASIDS ((1<= NR_AVAIL_ASIDS);
+   /*
+* If PCID is on, ASID-aware code paths put the ASID+1 into the PCID
+* bits.  This serves two purposes.  It prevents a nasty situation in
+* which PCID-unaware code saves CR3, loads some other value (with PCID
+* == 0), and then restores CR3, thus corrupting the TLB for ASID 0 if
+* the saved ASID was nonzero.  It also means that any bugs involving
+* loading a PCID-enabled CR3 with CR4.PCIDE off will trigger
+* deterministically.
+*/
+   return asid + 1;
+}
+
 struct pgd_t;
 static inline unsigned long build_cr3(pgd_t *pgd, u16 asid)
 {
if (static_cpu_has(X86_FEATURE_PCID)) {
-   VM_WARN_ON_ONCE(asid > NR_AVAIL_ASIDS);
-   return __sme_pa(pgd) | (asid + 1);
+   return __sme_pa(pgd) | kern_asid(asid);
} else {
VM_WARN_ON_ONCE(asid != 0);
return __sme_pa(pgd);
@@ -110,7 +115,8 @@ static inline unsigned long build_cr3(pg
 static inline unsigned long build_cr3_noflush(pgd_t *pgd, u16 asid)
 {
VM_WARN_ON_ONCE(asid > NR_AVAIL_ASIDS);
-   return __sme_pa(pgd) | (asid + 1) | CR3_NOFLUSH;
+   VM_WARN_ON_ONCE(!this_cpu_has(X86_FEATURE_PCID));
+   return __sme_pa(pgd) | kern_asid(asid) | CR3_NOFLUSH;
 }
 
 #ifdef CONFIG_PARAVIRT
_

[PATCH 21/30] x86, mm: put mmu-to-h/w ASID translation in one place

[Dave Hansen]

From: Dave Hansen 

We effectively have two ASID types:
1. The one stored in the mmu_context that goes from 0->5
2. The one we program into the hardware that goes from 1->6

Let's just put the +1 in a single place which gives us a
nice place to comment.  KAISER will also need to, given an
ASID, know which hardware ASID to flush for the userspace
mapping.

Signed-off-by: Dave Hansen 
Cc: Moritz Lipp 
Cc: Daniel Gruss 
Cc: Michael Schwarz 
Cc: Richard Fellner 
Cc: Andy Lutomirski 
Cc: Linus Torvalds 
Cc: Kees Cook 
Cc: Hugh Dickins 
Cc: x...@kernel.org
---

 b/arch/x86/include/asm/tlbflush.h |   30 ++
 1 file changed, 18 insertions(+), 12 deletions(-)

diff -puN arch/x86/include/asm/tlbflush.h~kaiser-pcid-pre-build-kern 
arch/x86/include/asm/tlbflush.h
--- a/arch/x86/include/asm/tlbflush.h~kaiser-pcid-pre-build-kern
2017-11-08 10:45:37.314681375 -0800
+++ b/arch/x86/include/asm/tlbflush.h   2017-11-08 10:45:37.317681375 -0800
@@ -86,21 +86,26 @@ static inline u64 inc_mm_tlb_gen(struct
  */
 #define NR_AVAIL_ASIDS ((1<= NR_AVAIL_ASIDS);
+   /*
+* If PCID is on, ASID-aware code paths put the ASID+1 into the PCID
+* bits.  This serves two purposes.  It prevents a nasty situation in
+* which PCID-unaware code saves CR3, loads some other value (with PCID
+* == 0), and then restores CR3, thus corrupting the TLB for ASID 0 if
+* the saved ASID was nonzero.  It also means that any bugs involving
+* loading a PCID-enabled CR3 with CR4.PCIDE off will trigger
+* deterministically.
+*/
+   return asid + 1;
+}
+
 struct pgd_t;
 static inline unsigned long build_cr3(pgd_t *pgd, u16 asid)
 {
if (static_cpu_has(X86_FEATURE_PCID)) {
-   VM_WARN_ON_ONCE(asid > NR_AVAIL_ASIDS);
-   return __sme_pa(pgd) | (asid + 1);
+   return __sme_pa(pgd) | kern_asid(asid);
} else {
VM_WARN_ON_ONCE(asid != 0);
return __sme_pa(pgd);
@@ -110,7 +115,8 @@ static inline unsigned long build_cr3(pg
 static inline unsigned long build_cr3_noflush(pgd_t *pgd, u16 asid)
 {
VM_WARN_ON_ONCE(asid > NR_AVAIL_ASIDS);
-   return __sme_pa(pgd) | (asid + 1) | CR3_NOFLUSH;
+   VM_WARN_ON_ONCE(!this_cpu_has(X86_FEATURE_PCID));
+   return __sme_pa(pgd) | kern_asid(asid) | CR3_NOFLUSH;
 }
 
 #ifdef CONFIG_PARAVIRT
_