[Tails-dev] What is *not* erased (after shutdown) with PAX_MEMORY_SANITIZE enabled?
Hi!
I'm working on Tails (https://tails.boum.org/).
Our threat model includes "someone breaks the door and you have 15
seconds to make your current Tails activity disappear". We assume that
this "someone" may be prepared to try and recover the memory of this
Tails system, e.g. via a cold boot attack.
I'm describing at the bottom of this email our current protection
against this scenario ("Current Tails implementation" section), in
case you're interested in more background; but for now I'll jump
directly to the questions I have for you.
We're considering dropping our current implementation, and relying on
PAX_MEMORY_SANITIZE instead; I know that PAX_MEMORY_SANITIZE was not
designed for this use case, but who knows: it might be good enough :)
I understand that the result won't achieve perfect protection, but our
current implementation doesn't either. I'd like to see numbers, but
it's hard for me to compare experimentally both approaches: our QA
about this relies on filling memory with a known pattern from
userspace, which works fine to measure the efficiency of our current
implementation, but of course PAX_MEMORY_SANITIZE will erase this
known pattern from memory… So for now I'll stick to the theoretical
level, and experimental measurements will come later.
If I got it right, with PAX_MEMORY_SANITIZE enabled, then:
* during the lifetime of a system, any memory allocated to
a userspace program that terminates is erased;
* during the lifetime of a system, any kernel memory that's
explicitly freed, e.g. with kfree(), is erased;
* on system shutdown, all processes are killed, and thus their memory
is erased.
So, what remains after system shutdown boils down to:
* kernel memory allocated and then freed, during the lifetime of the
system, through means that are not covered by PAX_MEMORY_SANITIZE:
is there any such thing? I'm interested in a (possibly incomplete)
list of these memory areas or allocation methods.
* kernel memory, that was still in use during shutdown, and that the
kernel does not explicitly free: again, is there any such thing?
* anything else?
I'm not a low-level person myself, so I'm happy to stand corrected,
and I'll probably need help from my team-mates (Cc'ed) to analyze your
answers :)
Current Tails implementation
Our current protection against this scenario is: when I unplug my
Tails USB stick, an emergency shutdown procedure is triggered, that
kexec's another kernel with some special command line parameter, and
then the initramfs erases all the memory that's available from
userspace. This ensures that even the memory that was kept in use by
the previous kernel (e.g. data in tmpfs) is overwritten at least once:
either by the new kernel re-using the same memory, or by the userspace
memory wipe process. Of course, some memory will still remain
unmodified, e.g. whatever data the kernel may have put aside out of
userspace's reach, without overwriting it. E.g. in our tests, a few
dozens of MiB are not erased by this process on a machine with 8 GiB
of RAM. Implementation details are documented there:
https://tails.boum.org/contribute/design/memory_erasure/
Cheers,
--
intrigeri
___
Tails-dev mailing list
Tails-dev@boum.org
https://mailman.boum.org/listinfo/tails-dev
To unsubscribe from this list, send an empty email to
tails-dev-unsubscr...@boum.org.
Re: [Tails-dev] What is *not* erased (after shutdown) with PAX_MEMORY_SANITIZE enabled?
On 2 Jan 2017 at 15:27, intrigeri wrote: Hi all, first of all, i've put spender on CC as an interested party (especially for the kexec vs. kmem hardening you're discussing in another thread). > We're considering dropping our current implementation, and relying on > PAX_MEMORY_SANITIZE instead; I know that PAX_MEMORY_SANITIZE was not > designed for this use case, but who knows: it might be good enough :) actually this isn't the first time this case has been raised and cleaning memory on shutdown has been on my todo list ever since, just not at a high enough priority. > I understand that the result won't achieve perfect protection, but our > current implementation doesn't either. I'd like to see numbers, but > it's hard for me to compare experimentally both approaches: our QA > about this relies on filling memory with a known pattern from > userspace, which works fine to measure the efficiency of our current > implementation, but of course PAX_MEMORY_SANITIZE will erase this > known pattern from memory… So for now I'll stick to the theoretical > level, and experimental measurements will come later. > > > If I got it right, with PAX_MEMORY_SANITIZE enabled, then: > > * during the lifetime of a system, any memory allocated to >a userspace program that terminates is erased; correct but don't forget that 'allocated to userland' means only anonymous memory pages that are mapped into userland memory, everything else (page cache holding file content, dentry/inode/etc metadata) is subject to different rules, see next. > * during the lifetime of a system, any kernel memory that's >explicitly freed, e.g. with kfree(), is erased; well, 'free' is a bit abstract concept to reason about at this level ;). the kernel has more than one way to allocate (and thus free) memory and SANITIZE doesn't (directly) apply to all of them. in practice most memory is managed by the buddy allocator and a few layers that build on top of it (slab, vmalloc, etc). the first (and original) layer where SANITIZE kicks in is the buddy allocator where memory is managed in groups of pages. this becomes important for the layers built on top of it since they can very well allocate/free memory at their level without every freeing back to the buddy allocator. for this reason a while ago Mathias Krause added support for sanitizing at the slab layer as well (slab manages C level objects, you can consider it as the kernel's equivalent of malloc) but for performance reasons not all slabs are sanitized by default (for tails you'll want to enable all of it i guess, see the config help on the boot time command line option or just patch the code to change the default which is probably safer for your case). > * on system shutdown, all processes are killed, and thus their memory >is erased. yes if (presumably :) you meant a clean shutdown (and subject to the above details). i'm just pointing it out because in the panic case you mentioned a clean shutdown may not be possible. > So, what remains after system shutdown boils down to: > > * kernel memory allocated and then freed, during the lifetime of the >system, through means that are not covered by PAX_MEMORY_SANITIZE: >is there any such thing? I'm interested in a (possibly incomplete) >list of these memory areas or allocation methods. while the buddy allocator acts as the lowest layer of memory manager for the kernel (so it sees everything as far as C code is concerned), there's also boot time memory allocated that is kept around and i'm pretty sure that some of it is never freed. there can also be arch-specific memory reservations that may need special handling (hugetlbfs, CMA, etc). > * kernel memory, that was still in use during shutdown, and that the >kernel does not explicitly free: again, is there any such thing? as hinted at above, there's lots of memory that is not process memory and that will be kept around as long as something uses that memory (or even when not). shutdown in linux isn't about undoing everything that happened before, it's simply just another function to execute that happens to never return. this leaves some memory in-use. just so you can get an idea, look at /proc/slabinfo from your shutdown script and count all the still allocated objects... now not all of them have sensitive information that you care about, but some definitely do. with all that said, let me comment on your current approach: > Current Tails implementation > > > Our current protection against this scenario is: when I unplug my > Tails USB stick, an emergency shutdown procedure is triggered, that > kexec's another kernel with some special command line parameter, and > then the initramfs erases all the memory that's available from > userspace. i'd like to interject here and point out a less than obvious sideeffect of SANITIZE (not really documented because the effect isn't relevant for the original purpose of SANITIZE): when the kernel tr
Re: [Tails-dev] What is *not* erased (after shutdown) with PAX_MEMORY_SANITIZE enabled?
> On 3 Jan 2017, at 21:40, PaX Team wrote: > > in other words, if you were to kexec into a SANITIZE enabled kernel, > you'd get your memory clearing for free automatically, earlier than > any initramfs would execute even and it'd cover most kernel memory > that the kernel ever cares about (or cared in its previous incarnation > at least). > > now this brings us to the other topic you raised about grsecurity's > KMEM hardening. technically it's not incompatible with kexec, so you > can re-enable kexec, however note that until some signed kexec mechanism > enters the kernel, it carries a risk of executing potentially malicious > kernels (but maybe that's not a problem in your use cases). perhaps > embedding or loading the kexec kernel from initramfs would get around > those concerns for good. Can you kexec from a running kernel into itself? If so, then this single use case could be enabled without opening a hole for arbitrary code. It would crash when it discovers the boot disk is missing, but by that time sanitize should have done its job. A. ___ Tails-dev mailing list Tails-dev@boum.org https://mailman.boum.org/listinfo/tails-dev To unsubscribe from this list, send an empty email to tails-dev-unsubscr...@boum.org.
Re: [Tails-dev] What is *not* erased (after shutdown) with PAX_MEMORY_SANITIZE enabled?
On 4 Jan 2017 at 1:17, Andrew Gallagher wrote: > > On 3 Jan 2017, at 21:40, PaX Team wrote: > > > > in other words, if you were to kexec into a SANITIZE enabled kernel, > > you'd get your memory clearing for free automatically, earlier than > > any initramfs would execute even and it'd cover most kernel memory > > that the kernel ever cares about (or cared in its previous incarnation > > at least). > > > > now this brings us to the other topic you raised about grsecurity's > > KMEM hardening. technically it's not incompatible with kexec, so you > > can re-enable kexec, however note that until some signed kexec mechanism > > enters the kernel, it carries a risk of executing potentially malicious > > kernels (but maybe that's not a problem in your use cases). perhaps > > embedding or loading the kexec kernel from initramfs would get around > > those concerns for good. > > Can you kexec from a running kernel into itself? if by 'itself' you mean the kernel image running in memory then the answer is 'no', or at least 'not without further surgery'. such surgery would involve among others: - not freeing the init code/data on boot - restoring the load-time content of writable kernel sections - simulating the early boot environment (memory map, firmware and/or boot loader provided data, etc) this is feasible but i'm not sure if it's worth the effort when you can just kexec a kernel, even the same one (loaded separately into memory of course). note also that i'm not really a kexec guy, so you should probably talk to the kernel maintainers as well... ___ Tails-dev mailing list Tails-dev@boum.org https://mailman.boum.org/listinfo/tails-dev To unsubscribe from this list, send an empty email to tails-dev-unsubscr...@boum.org.
Re: [Tails-dev] What is *not* erased (after shutdown) with PAX_MEMORY_SANITIZE enabled?
Hello everyone! Thanks for weighing in, PaX Team. (And thank you for the awesome you and spender do on kernel security!) To summarize, it seems that we have a couple different options to choose from: * Switch to a dedicated microkernel that does a memory wipe, and kexec into it. This could be something custom, or an enhancement of the preexisting solution. Advantages are that it will (probably) give us the most clean wipe, as we can reduce the amount of space the kernel takes up and drop all functionality that's not absolutely needed. On the negative side, it will require continuing to support a separate codeline that's not going to be reused elsewhere, limiting testing and development effort. It also requires us to reenable kexec functionality, which exposes a risk of code injection unless we get signed kexec support. * Rely on PAX_MEMORY_SANITIZE. This either takes the form of enhancing cleaning memory on shutdown, or kexec'ing into the same version of the kernel that's already running to rely on the buddy allocator clearing everything again. Definite pro in that it reduces the amount of code maintained downstream by the Tails team to ~zero. Cons are increased reliance on more complex functionality in the kernel, and potentially relying on a somewhat undocumented and unplanned functionality in the kernel. (It seems unlikely that it'll change any time without us noticing, but it's possible.) * Do it in userspace. Add functionality into the initramfs as necessary to wipe memory, and simply run an abbreviated shutdown. This lets us not have to deal with the potential for kexec's attack surface, and is probably some medium amount of code between the above two options. Downside is that it potentially is less reliable in terms of clearing memory than the other options, and is probably slower time-to-first-bit-erased than the other options. Does that seem like a fair summary to everyone? If so, which path seems the best for us to move forward on? -- Harlan Lieberman-Berg ~hlieberman ___ Tails-dev mailing list Tails-dev@boum.org https://mailman.boum.org/listinfo/tails-dev To unsubscribe from this list, send an empty email to tails-dev-unsubscr...@boum.org.
Re: [Tails-dev] What is *not* erased (after shutdown) with PAX_MEMORY_SANITIZE enabled?
Harlan Lieberman-Berg writes: > It also requires us to reenable kexec functionality, which exposes a > risk of code injection unless we get signed kexec support. I just checked the kernel, and it seems that signed kexec functionality was mainlined in 3.17. So, strike that from the list of problems with all the kexec dependent solutions. -- Harlan Lieberman-Berg ~hlieberman ___ Tails-dev mailing list Tails-dev@boum.org https://mailman.boum.org/listinfo/tails-dev To unsubscribe from this list, send an empty email to tails-dev-unsubscr...@boum.org.
Re: [Tails-dev] What is *not* erased (after shutdown) with PAX_MEMORY_SANITIZE enabled?
Hi everybody! Harlan Lieberman-Berg: > Thanks for weighing in, PaX Team. (And thank you for the awesome you > and spender do on kernel security!) +1 :) > To summarize, it seems that we have a couple different options to choose > from: > * Switch to a dedicated microkernel that does a memory wipe, and kexec > into it. This could be something custom, or an enhancement of the > preexisting solution. > Advantages are that it will (probably) give us the most clean wipe, as > we can reduce the amount of space the kernel takes up and drop all > functionality that's not absolutely needed. On the negative side, it > will require continuing to support a separate codeline that's not > going to be reused elsewhere, limiting testing and development > effort. It also requires us to reenable kexec functionality, which > exposes a risk of code injection unless we get signed kexec support. > * Rely on PAX_MEMORY_SANITIZE. This either takes the form of enhancing > cleaning memory on shutdown, or kexec'ing into the same version of the > kernel that's already running to rely on the buddy allocator clearing > everything again. > Definite pro in that it reduces the amount of code maintained > downstream by the Tails team to ~zero. Cons are increased reliance on > more complex functionality in the kernel, and potentially relying on a > somewhat undocumented and unplanned functionality in the kernel. (It > seems unlikely that it'll change any time without us noticing, but > it's possible.) > * Do it in userspace. Add functionality into the initramfs as > necessary to wipe memory, and simply run an abbreviated shutdown. > This lets us not have to deal with the potential for kexec's attack > surface, and is probably some medium amount of code between the above > two options. Downside is that it potentially is less reliable in > terms of clearing memory than the other options, and is probably > slower time-to-first-bit-erased than the other options. > Does that seem like a fair summary to everyone? It does seem like it to me. > If so, which path seems the best for us to move forward on? I prefer the two first options (mostly because the third one is essentially what we're already doing, and are trying to improve/replace). Not sure which one of those two yet, but the next thing to do in both cases is the same: make it possible to allow kexec without disabling GRKERNSEC_KMEM entirely. I don't know what configuration interface would be best: move kexec disabling out of GRKERNSEC_KMEM, to another kernel build-time config setting? Leave it as part of GRKERNSEC_KMEM, but add a sysctl to allow re-enabling kexec at runtime? pipacs, spender: what do you think? Cheers, -- intrigeri ___ Tails-dev mailing list Tails-dev@boum.org https://mailman.boum.org/listinfo/tails-dev To unsubscribe from this list, send an empty email to tails-dev-unsubscr...@boum.org.
