Re: [RFC PATCH v7 00/16] Add support for eXclusive Page Frame Ownership
On 1/10/19 1:09 PM, Khalid Aziz wrote: I am continuing to build on the work Juerg, Tycho and Julian have done on XPFO. After the last round of updates, we were seeing very significant performance penalties when stale TLB entries were flushed actively after an XPFO TLB update. Benchmark for measuring performance is kernel build using parallel make. To get full protection from ret2dir attackes, we must flush stale TLB entries. Performance penalty from flushing stale TLB entries goes up as the number of cores goes up. On a desktop class machine with only 4 cores, enabling TLB flush for stale entries causes system time for "make -j4" to go up by a factor of 2.614x but on a larger machine with 96 cores, system time with "make -j60" goes up by a factor of 26.366x! I have been working on reducing this performance penalty. I implemented a solution to reduce performance penalty and that has had large impact. When XPFO code flushes stale TLB entries, it does so for all CPUs on the system which may include CPUs that may not have any matching TLB entries or may never be scheduled to run the userspace task causing TLB flush. Problem is made worse by the fact that if number of entries being flushed exceeds tlb_single_page_flush_ceiling, it results in a full TLB flush on every CPU. A rogue process can launch a ret2dir attack only from a CPU that has dual mapping for its pages in physmap in its TLB. We can hence defer TLB flush on a CPU until a process that would have caused a TLB flush is scheduled on that CPU. I have added a cpumask to task_struct which is then used to post pending TLB flush on CPUs other than the one a process is running on. This cpumask is checked when a process migrates to a new CPU and TLB is flushed at that time. I measured system time for parallel make with unmodified 4.20 kernel, 4.20 with XPFO patches before this optimization and then again after applying this optimization. Here are the results: Hardware: 96-core Intel Xeon Platinum 8160 CPU @ 2.10GHz, 768 GB RAM make -j60 all 4.20915.183s 4.20+XPFO 24129.354s 26.366x 4.20+XPFO+Deferred flush1216.987s1.330xx Hardware: 4-core Intel Core i5-3550 CPU @ 3.30GHz, 8G RAM make -j4 all 4.20607.671s 4.20+XPFO 1588.646s 2.614x 4.20+XPFO+Deferred flush794.473s1.307xx 30+% overhead is still very high and there is room for improvement. Dave Hansen had suggested batch updating TLB entries and Tycho had created an initial implementation but I have not been able to get that to work correctly. I am still working on it and I suspect we will see a noticeable improvement in performance with that. In the code I added, I post a pending full TLB flush to all other CPUs even when number of TLB entries being flushed on current CPU does not exceed tlb_single_page_flush_ceiling. There has to be a better way to do this. I just haven't found an efficient way to implemented delayed limited TLB flush on other CPUs. I am not entirely sure if switch_mm_irqs_off() is indeed the right place to perform the pending TLB flush for a CPU. Any feedback on that will be very helpful. Delaying full TLB flushes on other CPUs seems to help tremendously, so if there is a better way to implement the same thing than what I have done in patch 16, I am open to ideas. Performance with this patch set is good enough to use these as starting point for further refinement before we merge it into main kernel, hence RFC. Since not flushing stale TLB entries creates a false sense of security, I would recommend making TLB flush mandatory and eliminate the "xpfotlbflush" kernel parameter (patch "mm, x86: omit TLB flushing by default for XPFO page table modifications"). What remains to be done beyond this patch series: 1. Performance improvements 2. Remove xpfotlbflush parameter 3. Re-evaluate the patch "arm64/mm: Add support for XPFO to swiotlb" from Juerg. I dropped it for now since swiotlb code for ARM has changed a lot in 4.20. 4. Extend the patch "xpfo, mm: Defer TLB flushes for non-current CPUs" to other architectures besides x86. - Juerg Haefliger (5): mm, x86: Add support for eXclusive Page Frame Ownership (XPFO) swiotlb: Map the buffer if it was unmapped by XPFO arm64/mm: Add support for XPFO arm64/mm, xpfo: temporarily map dcache regions lkdtm: Add test for XPFO Julian Stecklina (4): mm, x86: omit TLB flushing by default for XPFO page table modifications xpfo, mm: remove dependency on CONFIG_PAGE_EXTENSION xpfo, mm: optimize spinlock usage in xpfo_kunmap EXPERIMENTAL: xpfo, mm: optimize spin lock usage in xpfo_kmap Khalid Aziz (2): xpfo, mm: Fix hang when booting with "xpfotlbflush" xpfo, mm: Defer TLB flushes for non-current CPUs (x86 only) Tycho Andersen (5): mm: add MAP_HUGETLB support to vm_mmap x86: always set IF before
Re: [RFC PATCH v7 00/16] Add support for eXclusive Page Frame Ownership
On 1/16/19 7:56 AM, Julian Stecklina wrote: > Khalid Aziz writes: > >> I am continuing to build on the work Juerg, Tycho and Julian have done >> on XPFO. > > Awesome! > >> A rogue process can launch a ret2dir attack only from a CPU that has >> dual mapping for its pages in physmap in its TLB. We can hence defer >> TLB flush on a CPU until a process that would have caused a TLB flush >> is scheduled on that CPU. > > Assuming the attacker already has the ability to execute arbitrary code > in userspace, they can just create a second process and thus avoid the > TLB flush. Am I getting this wrong? No, you got it right. The patch I wrote closes the security hole when attack is launched from the same process but still leaves a window open when attack is launched from another process. I am working on figuring out how to close that hole while keeping the performance the same as it is now. Synchronous TLB flush across all cores is the most secure but performance impact is horrendous. -- Khalid pEpkey.asc Description: application/pgp-keys
Re: [RFC PATCH v7 00/16] Add support for eXclusive Page Frame Ownership
Khalid Aziz writes: > I am continuing to build on the work Juerg, Tycho and Julian have done > on XPFO. Awesome! > A rogue process can launch a ret2dir attack only from a CPU that has > dual mapping for its pages in physmap in its TLB. We can hence defer > TLB flush on a CPU until a process that would have caused a TLB flush > is scheduled on that CPU. Assuming the attacker already has the ability to execute arbitrary code in userspace, they can just create a second process and thus avoid the TLB flush. Am I getting this wrong? Julian
Re: [RFC PATCH v7 00/16] Add support for eXclusive Page Frame Ownership
On 1/10/19 1:09 PM, Khalid Aziz wrote: I am continuing to build on the work Juerg, Tycho and Julian have done on XPFO. After the last round of updates, we were seeing very significant performance penalties when stale TLB entries were flushed actively after an XPFO TLB update. Benchmark for measuring performance is kernel build using parallel make. To get full protection from ret2dir attackes, we must flush stale TLB entries. Performance penalty from flushing stale TLB entries goes up as the number of cores goes up. On a desktop class machine with only 4 cores, enabling TLB flush for stale entries causes system time for "make -j4" to go up by a factor of 2.614x but on a larger machine with 96 cores, system time with "make -j60" goes up by a factor of 26.366x! I have been working on reducing this performance penalty. I implemented a solution to reduce performance penalty and that has had large impact. When XPFO code flushes stale TLB entries, it does so for all CPUs on the system which may include CPUs that may not have any matching TLB entries or may never be scheduled to run the userspace task causing TLB flush. Problem is made worse by the fact that if number of entries being flushed exceeds tlb_single_page_flush_ceiling, it results in a full TLB flush on every CPU. A rogue process can launch a ret2dir attack only from a CPU that has dual mapping for its pages in physmap in its TLB. We can hence defer TLB flush on a CPU until a process that would have caused a TLB flush is scheduled on that CPU. I have added a cpumask to task_struct which is then used to post pending TLB flush on CPUs other than the one a process is running on. This cpumask is checked when a process migrates to a new CPU and TLB is flushed at that time. I measured system time for parallel make with unmodified 4.20 kernel, 4.20 with XPFO patches before this optimization and then again after applying this optimization. Here are the results: Hardware: 96-core Intel Xeon Platinum 8160 CPU @ 2.10GHz, 768 GB RAM make -j60 all 4.20915.183s 4.20+XPFO 24129.354s 26.366x 4.20+XPFO+Deferred flush1216.987s1.330xx Hardware: 4-core Intel Core i5-3550 CPU @ 3.30GHz, 8G RAM make -j4 all 4.20607.671s 4.20+XPFO 1588.646s 2.614x 4.20+XPFO+Deferred flush794.473s1.307xx 30+% overhead is still very high and there is room for improvement. Dave Hansen had suggested batch updating TLB entries and Tycho had created an initial implementation but I have not been able to get that to work correctly. I am still working on it and I suspect we will see a noticeable improvement in performance with that. In the code I added, I post a pending full TLB flush to all other CPUs even when number of TLB entries being flushed on current CPU does not exceed tlb_single_page_flush_ceiling. There has to be a better way to do this. I just haven't found an efficient way to implemented delayed limited TLB flush on other CPUs. I am not entirely sure if switch_mm_irqs_off() is indeed the right place to perform the pending TLB flush for a CPU. Any feedback on that will be very helpful. Delaying full TLB flushes on other CPUs seems to help tremendously, so if there is a better way to implement the same thing than what I have done in patch 16, I am open to ideas. Performance with this patch set is good enough to use these as starting point for further refinement before we merge it into main kernel, hence RFC. Since not flushing stale TLB entries creates a false sense of security, I would recommend making TLB flush mandatory and eliminate the "xpfotlbflush" kernel parameter (patch "mm, x86: omit TLB flushing by default for XPFO page table modifications"). What remains to be done beyond this patch series: 1. Performance improvements 2. Remove xpfotlbflush parameter 3. Re-evaluate the patch "arm64/mm: Add support for XPFO to swiotlb" from Juerg. I dropped it for now since swiotlb code for ARM has changed a lot in 4.20. 4. Extend the patch "xpfo, mm: Defer TLB flushes for non-current CPUs" to other architectures besides x86. - Juerg Haefliger (5): mm, x86: Add support for eXclusive Page Frame Ownership (XPFO) swiotlb: Map the buffer if it was unmapped by XPFO arm64/mm: Add support for XPFO arm64/mm, xpfo: temporarily map dcache regions lkdtm: Add test for XPFO Julian Stecklina (4): mm, x86: omit TLB flushing by default for XPFO page table modifications xpfo, mm: remove dependency on CONFIG_PAGE_EXTENSION xpfo, mm: optimize spinlock usage in xpfo_kunmap EXPERIMENTAL: xpfo, mm: optimize spin lock usage in xpfo_kmap Khalid Aziz (2): xpfo, mm: Fix hang when booting with "xpfotlbflush" xpfo, mm: Defer TLB flushes for non-current CPUs (x86 only) Tycho Andersen (5): mm: add MAP_HUGETLB support to vm_mmap x86: always set IF before
Re: [RFC PATCH v7 00/16] Add support for eXclusive Page Frame Ownership
On 1/11/19 2:06 PM, Andy Lutomirski wrote: > On Fri, Jan 11, 2019 at 12:42 PM Dave Hansen wrote: >> The second process could easily have the page's old TLB entry. It could abuse that entry as long as that CPU doesn't context switch (switch_mm_irqs_off()) or otherwise flush the TLB entry. >>> >>> That is an interesting scenario. Working through this scenario, physmap >>> TLB entry for a page is flushed on the local processor when the page is >>> allocated to userspace, in xpfo_alloc_pages(). When the userspace passes >>> page back into kernel, that page is mapped into kernel space using a va >>> from kmap pool in xpfo_kmap() which can be different for each new >>> mapping of the same page. The physical page is unmapped from kernel on >>> the way back from kernel to userspace by xpfo_kunmap(). So two processes >>> on different CPUs sharing same physical page might not be seeing the >>> same virtual address for that page while they are in the kernel, as long >>> as it is an address from kmap pool. ret2dir attack relies upon being >>> able to craft a predictable virtual address in the kernel physmap for a >>> physical page and redirect execution to that address. Does that sound right? >> >> All processes share one set of kernel page tables. Or, did your patches >> change that somehow that I missed? >> >> Since they share the page tables, they implicitly share kmap*() >> mappings. kmap_atomic() is not *used* by more than one CPU, but the >> mapping is accessible and at least exists for all processors. >> >> I'm basically assuming that any entry mapped in a shared page table is >> exploitable on any CPU regardless of where we logically *want* it to be >> used. >> >> > > We can, very easily, have kernel mappings that are private to a given > mm. Maybe this is useful here. > That sounds like an interesting idea. kmap mappings would be a good candidate for that. Those are temporary mappings and should only be valid for one process. -- Khalid pEpkey.asc Description: application/pgp-keys
Re: [RFC PATCH v7 00/16] Add support for eXclusive Page Frame Ownership
On 1/11/19 1:42 PM, Dave Hansen wrote: >>> The second process could easily have the page's old TLB entry. It could >>> abuse that entry as long as that CPU doesn't context switch >>> (switch_mm_irqs_off()) or otherwise flush the TLB entry. >> >> That is an interesting scenario. Working through this scenario, physmap >> TLB entry for a page is flushed on the local processor when the page is >> allocated to userspace, in xpfo_alloc_pages(). When the userspace passes >> page back into kernel, that page is mapped into kernel space using a va >> from kmap pool in xpfo_kmap() which can be different for each new >> mapping of the same page. The physical page is unmapped from kernel on >> the way back from kernel to userspace by xpfo_kunmap(). So two processes >> on different CPUs sharing same physical page might not be seeing the >> same virtual address for that page while they are in the kernel, as long >> as it is an address from kmap pool. ret2dir attack relies upon being >> able to craft a predictable virtual address in the kernel physmap for a >> physical page and redirect execution to that address. Does that sound right? > > All processes share one set of kernel page tables. Or, did your patches > change that somehow that I missed? > > Since they share the page tables, they implicitly share kmap*() > mappings. kmap_atomic() is not *used* by more than one CPU, but the > mapping is accessible and at least exists for all processors. > > I'm basically assuming that any entry mapped in a shared page table is > exploitable on any CPU regardless of where we logically *want* it to be > used. > > Ah, I see what you are saying. Virtual address on one processor is visible on the other processor as well and one process could communicate that va to the other process in some way so it could be exploited by the other process. This va is exploitable only between the kmap and matching kunmap but the window exists. I am trying to understand your scenario, so I can address it right. -- Khalid pEpkey.asc Description: application/pgp-keys
Re: [RFC PATCH v7 00/16] Add support for eXclusive Page Frame Ownership
On 1/10/19 5:44 PM, Andy Lutomirski wrote: > On Thu, Jan 10, 2019 at 3:07 PM Kees Cook wrote: >> >> On Thu, Jan 10, 2019 at 1:10 PM Khalid Aziz wrote: >>> I implemented a solution to reduce performance penalty and >>> that has had large impact. When XPFO code flushes stale TLB entries, >>> it does so for all CPUs on the system which may include CPUs that >>> may not have any matching TLB entries or may never be scheduled to >>> run the userspace task causing TLB flush. Problem is made worse by >>> the fact that if number of entries being flushed exceeds >>> tlb_single_page_flush_ceiling, it results in a full TLB flush on >>> every CPU. A rogue process can launch a ret2dir attack only from a >>> CPU that has dual mapping for its pages in physmap in its TLB. We >>> can hence defer TLB flush on a CPU until a process that would have >>> caused a TLB flush is scheduled on that CPU. I have added a cpumask >>> to task_struct which is then used to post pending TLB flush on CPUs >>> other than the one a process is running on. This cpumask is checked >>> when a process migrates to a new CPU and TLB is flushed at that >>> time. I measured system time for parallel make with unmodified 4.20 >>> kernel, 4.20 with XPFO patches before this optimization and then >>> again after applying this optimization. Here are the results: > > I wasn't cc'd on the patch, so I don't know the exact details. > > I'm assuming that "ret2dir" means that you corrupt the kernel into > using a direct-map page as its stack. If so, then I don't see why the > task in whose context the attack is launched needs to be the same > process as the one that has the page mapped for user access. You are right. More work is needed to refine delayed TLB flush to close this gap. > > My advice would be to attempt an entirely different optimization: try > to avoid putting pages *back* into the direct map when they're freed > until there is an actual need to use them for kernel purposes. I had thought about that but it turns out the performance impact happens on the initial allocation of the page and resulting TLB flushes, not from putting the pages back into direct map. The way we could benefit from not adding page back to direct map is if we change page allocation to prefer pages not in direct map. That way we incur the cost of TLB flushes initially but then satisfy multiple allocation requests after that from those "xpfo cost" free pages. More changes will be needed to pick which of these pages can be added back to direct map without degenerating into worst case scenario of a page bouncing constantly between this list of preferred pages and direct mapped pages. It started to get complex enough that I decided to put this in my back pocket and attempt simpler approaches first :) > > How are you handing page cache? Presumably MAP_SHARED PROT_WRITE > pages are still in the direct map so that IO works. > Since Juerg wrote the actual implementation of XPFO, he probably understands it better. XPFO tackles only the page allocation requests from userspace and does not touch page cache pages. -- Khalid pEpkey.asc Description: application/pgp-keys
Re: [RFC PATCH v7 00/16] Add support for eXclusive Page Frame Ownership
On Fri, Jan 11, 2019 at 12:42 PM Dave Hansen wrote: > > >> The second process could easily have the page's old TLB entry. It could > >> abuse that entry as long as that CPU doesn't context switch > >> (switch_mm_irqs_off()) or otherwise flush the TLB entry. > > > > That is an interesting scenario. Working through this scenario, physmap > > TLB entry for a page is flushed on the local processor when the page is > > allocated to userspace, in xpfo_alloc_pages(). When the userspace passes > > page back into kernel, that page is mapped into kernel space using a va > > from kmap pool in xpfo_kmap() which can be different for each new > > mapping of the same page. The physical page is unmapped from kernel on > > the way back from kernel to userspace by xpfo_kunmap(). So two processes > > on different CPUs sharing same physical page might not be seeing the > > same virtual address for that page while they are in the kernel, as long > > as it is an address from kmap pool. ret2dir attack relies upon being > > able to craft a predictable virtual address in the kernel physmap for a > > physical page and redirect execution to that address. Does that sound right? > > All processes share one set of kernel page tables. Or, did your patches > change that somehow that I missed? > > Since they share the page tables, they implicitly share kmap*() > mappings. kmap_atomic() is not *used* by more than one CPU, but the > mapping is accessible and at least exists for all processors. > > I'm basically assuming that any entry mapped in a shared page table is > exploitable on any CPU regardless of where we logically *want* it to be > used. > > We can, very easily, have kernel mappings that are private to a given mm. Maybe this is useful here.
Re: [RFC PATCH v7 00/16] Add support for eXclusive Page Frame Ownership
>> The second process could easily have the page's old TLB entry. It could >> abuse that entry as long as that CPU doesn't context switch >> (switch_mm_irqs_off()) or otherwise flush the TLB entry. > > That is an interesting scenario. Working through this scenario, physmap > TLB entry for a page is flushed on the local processor when the page is > allocated to userspace, in xpfo_alloc_pages(). When the userspace passes > page back into kernel, that page is mapped into kernel space using a va > from kmap pool in xpfo_kmap() which can be different for each new > mapping of the same page. The physical page is unmapped from kernel on > the way back from kernel to userspace by xpfo_kunmap(). So two processes > on different CPUs sharing same physical page might not be seeing the > same virtual address for that page while they are in the kernel, as long > as it is an address from kmap pool. ret2dir attack relies upon being > able to craft a predictable virtual address in the kernel physmap for a > physical page and redirect execution to that address. Does that sound right? All processes share one set of kernel page tables. Or, did your patches change that somehow that I missed? Since they share the page tables, they implicitly share kmap*() mappings. kmap_atomic() is not *used* by more than one CPU, but the mapping is accessible and at least exists for all processors. I'm basically assuming that any entry mapped in a shared page table is exploitable on any CPU regardless of where we logically *want* it to be used.
Re: [RFC PATCH v7 00/16] Add support for eXclusive Page Frame Ownership
Hi Dave, Thanks for looking at this and providing feedback. On 1/10/19 4:40 PM, Dave Hansen wrote: > First of all, thanks for picking this back up. It looks to be going in > a very positive direction! > > On 1/10/19 1:09 PM, Khalid Aziz wrote: >> I implemented a solution to reduce performance penalty and >> that has had large impact. When XPFO code flushes stale TLB entries, >> it does so for all CPUs on the system which may include CPUs that >> may not have any matching TLB entries or may never be scheduled to >> run the userspace task causing TLB flush. > ... >> A rogue process can launch a ret2dir attack only from a CPU that has >> dual mapping for its pages in physmap in its TLB. We can hence defer >> TLB flush on a CPU until a process that would have caused a TLB >> flush is scheduled on that CPU. > > This logic is a bit suspect to me. Imagine a situation where we have > two attacker processes: one which is causing page to go from > kernel->user (and be unmapped from the kernel) and a second process that > *was* accessing that page. > > The second process could easily have the page's old TLB entry. It could > abuse that entry as long as that CPU doesn't context switch > (switch_mm_irqs_off()) or otherwise flush the TLB entry. That is an interesting scenario. Working through this scenario, physmap TLB entry for a page is flushed on the local processor when the page is allocated to userspace, in xpfo_alloc_pages(). When the userspace passes page back into kernel, that page is mapped into kernel space using a va from kmap pool in xpfo_kmap() which can be different for each new mapping of the same page. The physical page is unmapped from kernel on the way back from kernel to userspace by xpfo_kunmap(). So two processes on different CPUs sharing same physical page might not be seeing the same virtual address for that page while they are in the kernel, as long as it is an address from kmap pool. ret2dir attack relies upon being able to craft a predictable virtual address in the kernel physmap for a physical page and redirect execution to that address. Does that sound right? Now what happens if only one of these cooperating processes allocates the page, places malicious payload on that page and passes the address of this page to the other process which can deduce physmap for the page through /proc and exploit the physmap entry for the page on its CPU. That must be the scenario you are referring to. > > As for where to flush the TLB... As you know, using synchronous IPIs is > obviously the most bulletproof from a mitigation perspective. If you > can batch the IPIs, you can get the overhead down, but you need to do > the flushes for a bunch of pages at once, which I think is what you were > exploring but haven't gotten working yet. > > Anything else you do will have *some* reduced mitigation value, which > isn't a deal-breaker (to me at least). Some ideas: Even without batched IPIs working reliably, I was able to measure the performance impact of this partially working solution. With just batched IPIs and no delayed TLB flushes, performance improved by a factor of 2. The 26x system time went down to 12x-13x but it was still too high and a non-starter. Combining batched IPI with delayed TLB flushes improved performance to about 1.1x as opposed to 1.33x with delayed TLB flush alone. Those numbers are very rough since the batching implementation is incomplete. > > Take a look at the SWITCH_TO_KERNEL_CR3 in head_64.S. Every time that > gets called, we've (potentially) just done a user->kernel transition and > might benefit from flushing the TLB. We're always doing a CR3 write (on > Meltdown-vulnerable hardware) and it can do a full TLB flush based on if > X86_CR3_PCID_NOFLUSH_BIT is set. So, when you need a TLB flush, you > would set a bit that ADJUST_KERNEL_CR3 would see on the next > user->kernel transition on *each* CPU. Potentially, multiple TLB > flushes could be coalesced this way. The downside of this is that > you're exposed to the old TLB entries if a flush is needed while you are > already *in* the kernel. > > You could also potentially do this from C code, like in the syscall > entry code, or in sensitive places, like when you're returning from a > guest after a VMEXIT in the kvm code. > Good suggestions. Thanks. I think benefit will be highest from batching TLB flushes. I see a lot of time consumed by full TLB flushes on other processors when local processor did only a limited TLB flush. I will continue to debug the batch TLB updates. -- Khalid pEpkey.asc Description: application/pgp-keys
Re: [RFC PATCH v7 00/16] Add support for eXclusive Page Frame Ownership
On Thu, Jan 10, 2019 at 03:40:04PM -0800, Dave Hansen wrote: > Anything else you do will have *some* reduced mitigation value, which > isn't a deal-breaker (to me at least). Some ideas: > > Take a look at the SWITCH_TO_KERNEL_CR3 in head_64.S. Every time that > gets called, we've (potentially) just done a user->kernel transition and > might benefit from flushing the TLB. We're always doing a CR3 write (on > Meltdown-vulnerable hardware) and it can do a full TLB flush based on if > X86_CR3_PCID_NOFLUSH_BIT is set. So, when you need a TLB flush, you > would set a bit that ADJUST_KERNEL_CR3 would see on the next > user->kernel transition on *each* CPU. Potentially, multiple TLB > flushes could be coalesced this way. The downside of this is that > you're exposed to the old TLB entries if a flush is needed while you are > already *in* the kernel. I would really prefer not to depend on the PTI crud for new stuff. We really want to get rid of that code on unaffected CPUs.
Re: [RFC PATCH v7 00/16] Add support for eXclusive Page Frame Ownership
On Thu, Jan 10, 2019 at 3:07 PM Kees Cook wrote: > > On Thu, Jan 10, 2019 at 1:10 PM Khalid Aziz wrote: > > I implemented a solution to reduce performance penalty and > > that has had large impact. When XPFO code flushes stale TLB entries, > > it does so for all CPUs on the system which may include CPUs that > > may not have any matching TLB entries or may never be scheduled to > > run the userspace task causing TLB flush. Problem is made worse by > > the fact that if number of entries being flushed exceeds > > tlb_single_page_flush_ceiling, it results in a full TLB flush on > > every CPU. A rogue process can launch a ret2dir attack only from a > > CPU that has dual mapping for its pages in physmap in its TLB. We > > can hence defer TLB flush on a CPU until a process that would have > > caused a TLB flush is scheduled on that CPU. I have added a cpumask > > to task_struct which is then used to post pending TLB flush on CPUs > > other than the one a process is running on. This cpumask is checked > > when a process migrates to a new CPU and TLB is flushed at that > > time. I measured system time for parallel make with unmodified 4.20 > > kernel, 4.20 with XPFO patches before this optimization and then > > again after applying this optimization. Here are the results: I wasn't cc'd on the patch, so I don't know the exact details. I'm assuming that "ret2dir" means that you corrupt the kernel into using a direct-map page as its stack. If so, then I don't see why the task in whose context the attack is launched needs to be the same process as the one that has the page mapped for user access. My advice would be to attempt an entirely different optimization: try to avoid putting pages *back* into the direct map when they're freed until there is an actual need to use them for kernel purposes. How are you handing page cache? Presumably MAP_SHARED PROT_WRITE pages are still in the direct map so that IO works.
Re: [RFC PATCH v7 00/16] Add support for eXclusive Page Frame Ownership
Thanks for looking this over. On 1/10/19 4:07 PM, Kees Cook wrote: > On Thu, Jan 10, 2019 at 1:10 PM Khalid Aziz wrote: >> I implemented a solution to reduce performance penalty and >> that has had large impact. When XPFO code flushes stale TLB entries, >> it does so for all CPUs on the system which may include CPUs that >> may not have any matching TLB entries or may never be scheduled to >> run the userspace task causing TLB flush. Problem is made worse by >> the fact that if number of entries being flushed exceeds >> tlb_single_page_flush_ceiling, it results in a full TLB flush on >> every CPU. A rogue process can launch a ret2dir attack only from a >> CPU that has dual mapping for its pages in physmap in its TLB. We >> can hence defer TLB flush on a CPU until a process that would have >> caused a TLB flush is scheduled on that CPU. I have added a cpumask >> to task_struct which is then used to post pending TLB flush on CPUs >> other than the one a process is running on. This cpumask is checked >> when a process migrates to a new CPU and TLB is flushed at that >> time. I measured system time for parallel make with unmodified 4.20 >> kernel, 4.20 with XPFO patches before this optimization and then >> again after applying this optimization. Here are the results: >> >> Hardware: 96-core Intel Xeon Platinum 8160 CPU @ 2.10GHz, 768 GB RAM >> make -j60 all >> >> 4.20915.183s >> 4.20+XPFO 24129.354s 26.366x >> 4.20+XPFO+Deferred flush1216.987s1.330xx >> >> >> Hardware: 4-core Intel Core i5-3550 CPU @ 3.30GHz, 8G RAM >> make -j4 all >> >> 4.20607.671s >> 4.20+XPFO 1588.646s 2.614x >> 4.20+XPFO+Deferred flush794.473s1.307xx > > Well that's an impressive improvement! Nice work. :) > > (Are the cpumask improvements possible to be extended to other TLB > flushing needs? i.e. could there be other performance gains with that > code even for a non-XPFO system?) It may be usable for other situations as well but I have not given it any thought yet. I will take a look. > >> 30+% overhead is still very high and there is room for improvement. >> Dave Hansen had suggested batch updating TLB entries and Tycho had >> created an initial implementation but I have not been able to get >> that to work correctly. I am still working on it and I suspect we >> will see a noticeable improvement in performance with that. In the >> code I added, I post a pending full TLB flush to all other CPUs even >> when number of TLB entries being flushed on current CPU does not >> exceed tlb_single_page_flush_ceiling. There has to be a better way >> to do this. I just haven't found an efficient way to implemented >> delayed limited TLB flush on other CPUs. >> >> I am not entirely sure if switch_mm_irqs_off() is indeed the right >> place to perform the pending TLB flush for a CPU. Any feedback on >> that will be very helpful. Delaying full TLB flushes on other CPUs >> seems to help tremendously, so if there is a better way to implement >> the same thing than what I have done in patch 16, I am open to >> ideas. > > Dave, Andy, Ingo, Thomas, does anyone have time to look this over? > >> Performance with this patch set is good enough to use these as >> starting point for further refinement before we merge it into main >> kernel, hence RFC. >> >> Since not flushing stale TLB entries creates a false sense of >> security, I would recommend making TLB flush mandatory and eliminate >> the "xpfotlbflush" kernel parameter (patch "mm, x86: omit TLB >> flushing by default for XPFO page table modifications"). > > At this point, yes, that does seem to make sense. > >> What remains to be done beyond this patch series: >> >> 1. Performance improvements >> 2. Remove xpfotlbflush parameter >> 3. Re-evaluate the patch "arm64/mm: Add support for XPFO to swiotlb" >>from Juerg. I dropped it for now since swiotlb code for ARM has >>changed a lot in 4.20. >> 4. Extend the patch "xpfo, mm: Defer TLB flushes for non-current >>CPUs" to other architectures besides x86. > > This seems like a good plan. > > I've put this series in one of my tree so that 0day will find it and > grind tests... > https://git.kernel.org/pub/scm/linux/kernel/git/kees/linux.git/log/?h=kspp/xpfo/v7 Thanks for doing that! -- Khalid pEpkey.asc Description: application/pgp-keys
Re: [RFC PATCH v7 00/16] Add support for eXclusive Page Frame Ownership
First of all, thanks for picking this back up. It looks to be going in a very positive direction! On 1/10/19 1:09 PM, Khalid Aziz wrote: > I implemented a solution to reduce performance penalty and > that has had large impact. When XPFO code flushes stale TLB entries, > it does so for all CPUs on the system which may include CPUs that > may not have any matching TLB entries or may never be scheduled to > run the userspace task causing TLB flush. ... > A rogue process can launch a ret2dir attack only from a CPU that has > dual mapping for its pages in physmap in its TLB. We can hence defer > TLB flush on a CPU until a process that would have caused a TLB > flush is scheduled on that CPU. This logic is a bit suspect to me. Imagine a situation where we have two attacker processes: one which is causing page to go from kernel->user (and be unmapped from the kernel) and a second process that *was* accessing that page. The second process could easily have the page's old TLB entry. It could abuse that entry as long as that CPU doesn't context switch (switch_mm_irqs_off()) or otherwise flush the TLB entry. As for where to flush the TLB... As you know, using synchronous IPIs is obviously the most bulletproof from a mitigation perspective. If you can batch the IPIs, you can get the overhead down, but you need to do the flushes for a bunch of pages at once, which I think is what you were exploring but haven't gotten working yet. Anything else you do will have *some* reduced mitigation value, which isn't a deal-breaker (to me at least). Some ideas: Take a look at the SWITCH_TO_KERNEL_CR3 in head_64.S. Every time that gets called, we've (potentially) just done a user->kernel transition and might benefit from flushing the TLB. We're always doing a CR3 write (on Meltdown-vulnerable hardware) and it can do a full TLB flush based on if X86_CR3_PCID_NOFLUSH_BIT is set. So, when you need a TLB flush, you would set a bit that ADJUST_KERNEL_CR3 would see on the next user->kernel transition on *each* CPU. Potentially, multiple TLB flushes could be coalesced this way. The downside of this is that you're exposed to the old TLB entries if a flush is needed while you are already *in* the kernel. You could also potentially do this from C code, like in the syscall entry code, or in sensitive places, like when you're returning from a guest after a VMEXIT in the kvm code.
Re: [RFC PATCH v7 00/16] Add support for eXclusive Page Frame Ownership
On Thu, Jan 10, 2019 at 1:10 PM Khalid Aziz wrote: > I implemented a solution to reduce performance penalty and > that has had large impact. When XPFO code flushes stale TLB entries, > it does so for all CPUs on the system which may include CPUs that > may not have any matching TLB entries or may never be scheduled to > run the userspace task causing TLB flush. Problem is made worse by > the fact that if number of entries being flushed exceeds > tlb_single_page_flush_ceiling, it results in a full TLB flush on > every CPU. A rogue process can launch a ret2dir attack only from a > CPU that has dual mapping for its pages in physmap in its TLB. We > can hence defer TLB flush on a CPU until a process that would have > caused a TLB flush is scheduled on that CPU. I have added a cpumask > to task_struct which is then used to post pending TLB flush on CPUs > other than the one a process is running on. This cpumask is checked > when a process migrates to a new CPU and TLB is flushed at that > time. I measured system time for parallel make with unmodified 4.20 > kernel, 4.20 with XPFO patches before this optimization and then > again after applying this optimization. Here are the results: > > Hardware: 96-core Intel Xeon Platinum 8160 CPU @ 2.10GHz, 768 GB RAM > make -j60 all > > 4.20915.183s > 4.20+XPFO 24129.354s 26.366x > 4.20+XPFO+Deferred flush1216.987s1.330xx > > > Hardware: 4-core Intel Core i5-3550 CPU @ 3.30GHz, 8G RAM > make -j4 all > > 4.20607.671s > 4.20+XPFO 1588.646s 2.614x > 4.20+XPFO+Deferred flush794.473s1.307xx Well that's an impressive improvement! Nice work. :) (Are the cpumask improvements possible to be extended to other TLB flushing needs? i.e. could there be other performance gains with that code even for a non-XPFO system?) > 30+% overhead is still very high and there is room for improvement. > Dave Hansen had suggested batch updating TLB entries and Tycho had > created an initial implementation but I have not been able to get > that to work correctly. I am still working on it and I suspect we > will see a noticeable improvement in performance with that. In the > code I added, I post a pending full TLB flush to all other CPUs even > when number of TLB entries being flushed on current CPU does not > exceed tlb_single_page_flush_ceiling. There has to be a better way > to do this. I just haven't found an efficient way to implemented > delayed limited TLB flush on other CPUs. > > I am not entirely sure if switch_mm_irqs_off() is indeed the right > place to perform the pending TLB flush for a CPU. Any feedback on > that will be very helpful. Delaying full TLB flushes on other CPUs > seems to help tremendously, so if there is a better way to implement > the same thing than what I have done in patch 16, I am open to > ideas. Dave, Andy, Ingo, Thomas, does anyone have time to look this over? > Performance with this patch set is good enough to use these as > starting point for further refinement before we merge it into main > kernel, hence RFC. > > Since not flushing stale TLB entries creates a false sense of > security, I would recommend making TLB flush mandatory and eliminate > the "xpfotlbflush" kernel parameter (patch "mm, x86: omit TLB > flushing by default for XPFO page table modifications"). At this point, yes, that does seem to make sense. > What remains to be done beyond this patch series: > > 1. Performance improvements > 2. Remove xpfotlbflush parameter > 3. Re-evaluate the patch "arm64/mm: Add support for XPFO to swiotlb" >from Juerg. I dropped it for now since swiotlb code for ARM has >changed a lot in 4.20. > 4. Extend the patch "xpfo, mm: Defer TLB flushes for non-current >CPUs" to other architectures besides x86. This seems like a good plan. I've put this series in one of my tree so that 0day will find it and grind tests... https://git.kernel.org/pub/scm/linux/kernel/git/kees/linux.git/log/?h=kspp/xpfo/v7 Thanks! -- Kees Cook
[RFC PATCH v7 00/16] Add support for eXclusive Page Frame Ownership
I am continuing to build on the work Juerg, Tycho and Julian have done on XPFO. After the last round of updates, we were seeing very significant performance penalties when stale TLB entries were flushed actively after an XPFO TLB update. Benchmark for measuring performance is kernel build using parallel make. To get full protection from ret2dir attackes, we must flush stale TLB entries. Performance penalty from flushing stale TLB entries goes up as the number of cores goes up. On a desktop class machine with only 4 cores, enabling TLB flush for stale entries causes system time for "make -j4" to go up by a factor of 2.614x but on a larger machine with 96 cores, system time with "make -j60" goes up by a factor of 26.366x! I have been working on reducing this performance penalty. I implemented a solution to reduce performance penalty and that has had large impact. When XPFO code flushes stale TLB entries, it does so for all CPUs on the system which may include CPUs that may not have any matching TLB entries or may never be scheduled to run the userspace task causing TLB flush. Problem is made worse by the fact that if number of entries being flushed exceeds tlb_single_page_flush_ceiling, it results in a full TLB flush on every CPU. A rogue process can launch a ret2dir attack only from a CPU that has dual mapping for its pages in physmap in its TLB. We can hence defer TLB flush on a CPU until a process that would have caused a TLB flush is scheduled on that CPU. I have added a cpumask to task_struct which is then used to post pending TLB flush on CPUs other than the one a process is running on. This cpumask is checked when a process migrates to a new CPU and TLB is flushed at that time. I measured system time for parallel make with unmodified 4.20 kernel, 4.20 with XPFO patches before this optimization and then again after applying this optimization. Here are the results: Hardware: 96-core Intel Xeon Platinum 8160 CPU @ 2.10GHz, 768 GB RAM make -j60 all 4.20915.183s 4.20+XPFO 24129.354s 26.366x 4.20+XPFO+Deferred flush1216.987s1.330xx Hardware: 4-core Intel Core i5-3550 CPU @ 3.30GHz, 8G RAM make -j4 all 4.20607.671s 4.20+XPFO 1588.646s 2.614x 4.20+XPFO+Deferred flush794.473s1.307xx 30+% overhead is still very high and there is room for improvement. Dave Hansen had suggested batch updating TLB entries and Tycho had created an initial implementation but I have not been able to get that to work correctly. I am still working on it and I suspect we will see a noticeable improvement in performance with that. In the code I added, I post a pending full TLB flush to all other CPUs even when number of TLB entries being flushed on current CPU does not exceed tlb_single_page_flush_ceiling. There has to be a better way to do this. I just haven't found an efficient way to implemented delayed limited TLB flush on other CPUs. I am not entirely sure if switch_mm_irqs_off() is indeed the right place to perform the pending TLB flush for a CPU. Any feedback on that will be very helpful. Delaying full TLB flushes on other CPUs seems to help tremendously, so if there is a better way to implement the same thing than what I have done in patch 16, I am open to ideas. Performance with this patch set is good enough to use these as starting point for further refinement before we merge it into main kernel, hence RFC. Since not flushing stale TLB entries creates a false sense of security, I would recommend making TLB flush mandatory and eliminate the "xpfotlbflush" kernel parameter (patch "mm, x86: omit TLB flushing by default for XPFO page table modifications"). What remains to be done beyond this patch series: 1. Performance improvements 2. Remove xpfotlbflush parameter 3. Re-evaluate the patch "arm64/mm: Add support for XPFO to swiotlb" from Juerg. I dropped it for now since swiotlb code for ARM has changed a lot in 4.20. 4. Extend the patch "xpfo, mm: Defer TLB flushes for non-current CPUs" to other architectures besides x86. - Juerg Haefliger (5): mm, x86: Add support for eXclusive Page Frame Ownership (XPFO) swiotlb: Map the buffer if it was unmapped by XPFO arm64/mm: Add support for XPFO arm64/mm, xpfo: temporarily map dcache regions lkdtm: Add test for XPFO Julian Stecklina (4): mm, x86: omit TLB flushing by default for XPFO page table modifications xpfo, mm: remove dependency on CONFIG_PAGE_EXTENSION xpfo, mm: optimize spinlock usage in xpfo_kunmap EXPERIMENTAL: xpfo, mm: optimize spin lock usage in xpfo_kmap Khalid Aziz (2): xpfo, mm: Fix hang when booting with "xpfotlbflush" xpfo, mm: Defer TLB flushes for non-current CPUs (x86 only) Tycho Andersen (5): mm: add MAP_HUGETLB support to vm_mmap x86: always set IF before oopsing from page fault xpfo: add primitives for mapping
