Re: amd64: kernel aslr support

[Maxime Villard]

Le 17/01/2018 à 21:48, Thomas Klausner a écrit :

On Wed, Jan 17, 2018 at 08:01:19AM +0100, Maxime Villard wrote:

Why does GENERIC_KASLR disable KDTRACE_HOOKS? Is this necessary, or
are KDTRACE_HOOKS lowering the security somehow?


In fact, it's because KDTRACE_HOOKS wants to parse one CTF section; but with
our implementation we have several of them, and KDTRACE_HOOKS does not
handle that.


Chuck's patch for better ZFS/DTRACE support includes parsing of
multiple CTF sections, intended for use with kernel modules (so that
each kernel module can include its CTF). Will that be sufficient for
KASLR?


I guess so. But I remember there was also another issue, for some reason the
multiple CTF sections had a very huge size, and I didn't understand why. But
it can be re-investigated once the new DTRACE support is in the tree.

Maxime

Re: amd64: kernel aslr support

[Thomas Klausner]

On Wed, Jan 17, 2018 at 08:01:19AM +0100, Maxime Villard wrote:
> Or, you can dump the location of the segments by loading a kernel module
> that calls the following function
...
> It will give you the addresses of the pages containing the kernel sections.
> Note however that the sections may not start exactly at the printed addresses,
> they are shifted.

Thanks, that worked. I've attached the kernel module I wrote (put it
in src/sys/modules/aslr_test).

Here's the output:

GENERIC:
Segment 0 (text): va=0x8020 size=12582912
Segment 1 (rodata): va=0x80e0 size=6291456
Segment 2 (data): va=0x8140 size=8388608

KASLR, boot 1
Segment 0 (text): va=0xc340 size=2097152
Segment 1 (rodata): va=0xfa0ec000 size=4096
Segment 2 (rodata): va=0xf5e0 size=2097152
Segment 3 (rodata): va=0xbd80 size=2097152
Segment 4 (rodata): va=0x9fb39000 size=4096
Segment 5 (rodata): va=0xa12a6000 size=4096
Segment 6 (rodata): va=0xd7872000 size=4096
Segment 7 (rodata): va=0xe37e2000 size=4096
Segment 8 (rodata): va=0xa4f15000 size=4096
Segment 9 (rodata): va=0xcb613000 size=4096
Segment 10 (rodata): va=0xbf82a000 size=4096
Segment 11 (rodata): va=0x95414000 size=4096
Segment 12 (rodata): va=0xbf8f1000 size=4096
Segment 13 (rodata): va=0xa0fa2000 size=4096
Segment 14 (rodata): va=0x835b5000 size=4096
Segment 15 (data): va=0xf8c0 size=2097152
Segment 16 (data): va=0x8d00 size=2097152
Segment 17 (data): va=0xe300 size=2097152
Segment 18 (data): va=0xa5e0 size=2097152
Segment 19 (text): va=0xff80 size=2097152
Segment 20 (text): va=0xc3a0 size=2097152
Segment 21 (text): va=0x9d40 size=2097152
Segment 22 (text): va=0xc360 size=2097152
Segment 23 (text): va=0xca80 size=2097152
Segment 24 (text): va=0xe8e0 size=2097152
Segment 25 (text): va=0xc5e0 size=2097152
Segment 26 (text): va=0xf6a0 size=2097152
Segment 27 (text): va=0xec80 size=2097152
Segment 28 (text): va=0xc660 size=2097152
Segment 29 (text): va=0x85a0 size=2097152
Segment 30 (rodata): va=0xdb00 size=2097152
Segment 31 (rodata): va=0xfac0 size=2097152
Segment 32 (rodata): va=0x8e60 size=2097152
Segment 33 (rodata): va=0xb840 size=2097152
Segment 34 (rodata): va=0x9440 size=2097152

KASLR, boot 2
Segment 0 (text): va=0xb700 size=2097152
Segment 1 (rodata): va=0xc1b36000 size=4096
Segment 2 (rodata): va=0xc740 size=2097152
Segment 3 (rodata): va=0xe7e0 size=2097152
Segment 4 (rodata): va=0x93453000 size=4096
Segment 5 (rodata): va=0xa426 size=4096
Segment 6 (rodata): va=0x862c7000 size=4096
Segment 7 (rodata): va=0xd8681000 size=4096
Segment 8 (rodata): va=0xaec34000 size=4096
Segment 9 (rodata): va=0xdba5e000 size=4096
Segment 10 (rodata): va=0x8e14c000 size=4096
Segment 11 (rodata): va=0xb2cbc000 size=4096
Segment 12 (rodata): va=0xa5a5e000 size=4096
Segment 13 (rodata): va=0xdc716000 size=4096
Segment 14 (rodata): va=0x96f8b000 size=4096
Segment 15 (data): va=0xd1e0 size=2097152
Segment 16 (data): va=0xcec0 size=2097152
Segment 17 (data): va=0xec40 size=2097152
Segment 18 (data): va=0xc3e0 size=2097152
Segment 19 (text): va=0x92c0 size=2097152
Segment 20 (text): va=0x8c20 size=2097152
Segment 21 (text): va=0x9a20 size=2097152
Segment 22 (text): va=0xa460 size=2097152
Segment 23 (text): va=0xe9a0 size=2097152
Segment 24 (text): va=0xab60 size=2097152
Segment 25 (text): va=0xd3c0 size=2097152
Segment 26 (text): va=0x8340 size=2097152
Segment 27 (text): va=0xf120 size=2097152
Segment 28 (text): va=0xd240 size=2097152
Segment 29 (text): va=0xc900 size=2097152
Segment 30 (rodata): va=0xaf80 size=2097152
Segment 31 (rodata): va=0xacc0 size=2097152
Segment 32 (rodata): va=0xae60 size=2097152
Segment 33 (rodata): va=0xf160 size=2097152
Segment 34 (rodata): va=0xad00 size=2097152

> > Why does GENERIC_KASLR disable KDTRACE_HOOKS? Is this necessary, or
> > are KDTRACE_HOOKS lowering the security somehow?
> 
> In fact, it's because KDTRACE_HOOKS wants to parse one CTF section; but with
> our implementation we have several of them, and KDTRACE_HOOKS does not
> handle that.

Chuck's patch for better ZFS/DTRACE support includes parsing of
multiple CTF sections, intended for use with kernel modules (so that
each kernel module can include its CTF). Will that be sufficient for
KASLR?

Thanks,
 Thomas
# $NetBSD: Makefile,v 1.1 

Re: amd64: kernel aslr support

[Maxime Villard]

Le 16/01/2018 à 23:54, Thomas Klausner a écrit :

Hi!

On Wed, Nov 15, 2017 at 07:40:55PM +0100, Maxime Villard wrote:

Le 14/11/2017 à 15:43, Maxime Villard a écrit :

The size and number of these blocks is controlled by the split-by-file
parameter in Makefile.amd64. Right now it is set to 2MB, which produces a
kernel with ~23 allocatable (ie useful at runtime) sections, which is a third
of the total number supported (BTSPACE_NSEGS = 64). I will probably reduce
this parameter a bit in the future, to 1.5MB, or even 1MB.


Actually I just did it. So now it's 1MB (better security), physically shifted
by the prekern (better entropy), and mapped with large pages (better
performance). And along the way it mostly mitigates TLB cache attacks.

This is still wip but feel free to test, as always,


I've tried out the instructions at
http://m00nbsd.net/542a5cfd448aaf7db7adcadce74123d2.html and they
worked fine for me. Thank you!

I have a couple questions:

How can I check (after booting) if the kernel is using ASLR properly?


Well, you could load a kernel module and print the address of some global
variable; it should be different upon each reboot.

Or, you can dump the location of the segments by loading a kernel module
that calls the following function (written from memory):

void DumpSegments(void)
{
extern struct bootspace bootspace;
size_t i;

char names[4] = {
[BTSEG_NONE] = "none",
[BTSEG_TEXT] = "text",
[BTSEG_RODATA] = "rodata",
[BTSEG_DATA] = "data"
};

for (i = 0; i < BTSPACE_NSEGS; i++) {
if (bootspace.segs[i].type == BTSEG_NONE) {
continue;
}
printf("Segment %zu (%s): va=%p size=%zu\n", i,
names[bootspace.segs[i].type],
(void *)bootspace.segs[i].va,
bootspace.segs[i].sz);
}
}

It will give you the addresses of the pages containing the kernel sections.
Note however that the sections may not start exactly at the printed addresses,
they are shifted.


Why does GENERIC_KASLR disable KDTRACE_HOOKS? Is this necessary, or
are KDTRACE_HOOKS lowering the security somehow?


In fact, it's because KDTRACE_HOOKS wants to parse one CTF section; but with
our implementation we have several of them, and KDTRACE_HOOKS does not
handle that.

There is also GPROF (in kernel mode) that does not work with several text
sections, I'll have to blacklist/fix it too. It appears to be greatly broken,
by the way.

But basically, KASLR should work as-is for everything else.

Maxime

Re: amd64: kernel aslr support

[Thomas Klausner]

Hi!

On Wed, Nov 15, 2017 at 07:40:55PM +0100, Maxime Villard wrote:
> Le 14/11/2017 à 15:43, Maxime Villard a écrit :
> > The size and number of these blocks is controlled by the split-by-file
> > parameter in Makefile.amd64. Right now it is set to 2MB, which produces a
> > kernel with ~23 allocatable (ie useful at runtime) sections, which is a 
> > third
> > of the total number supported (BTSPACE_NSEGS = 64). I will probably reduce
> > this parameter a bit in the future, to 1.5MB, or even 1MB.
> 
> Actually I just did it. So now it's 1MB (better security), physically shifted
> by the prekern (better entropy), and mapped with large pages (better
> performance). And along the way it mostly mitigates TLB cache attacks.
> 
> This is still wip but feel free to test, as always,

I've tried out the instructions at
http://m00nbsd.net/542a5cfd448aaf7db7adcadce74123d2.html and they
worked fine for me. Thank you!

I have a couple questions:

How can I check (after booting) if the kernel is using ASLR properly?

Why does GENERIC_KASLR disable KDTRACE_HOOKS? Is this necessary, or
are KDTRACE_HOOKS lowering the security somehow?

Thanks,
 Thomas

Re: amd64: kernel aslr support

[Maxime Villard]

Le 14/11/2017 à 15:43, Maxime Villard a écrit :

The size and number of these blocks is controlled by the split-by-file
parameter in Makefile.amd64. Right now it is set to 2MB, which produces a
kernel with ~23 allocatable (ie useful at runtime) sections, which is a third
of the total number supported (BTSPACE_NSEGS = 64). I will probably reduce
this parameter a bit in the future, to 1.5MB, or even 1MB.


Actually I just did it. So now it's 1MB (better security), physically shifted
by the prekern (better entropy), and mapped with large pages (better
performance). And along the way it mostly mitigates TLB cache attacks.

This is still wip but feel free to test, as always,
Maxime

Re: amd64: kernel aslr support

[Thor Lancelot Simon]

On Tue, Nov 14, 2017 at 04:04:39PM +, Christos Zoulas wrote:
> In article <5cee5471-dc6f-db16-8914-75ad5ad15...@m00nbsd.net>,
> Maxime Villard   wrote:
> >
> >All of that leaves us with about the most advanced KASLR implementation
> >available out there. There are ways to improve it even more, but you'll have
> >to wait a few weeks for that.
> >
> >If you want to try it out you need to make sure you have the latest versions
> >of GENERIC_KASLR / prekern / bootloader. The instructions are still here [2],
> >and haven't changed.
> 
> Very nicely done!

Seriously!  Nice work!

-- 
  Thor Lancelot Simont...@panix.com
 "The two most common variations translate as follows:
illegitimi non carborundum = the unlawful are not silicon carbide
illegitimis non carborundum = the unlawful don't have silicon carbide."

Re: amd64: kernel aslr support

[Christos Zoulas]

In article <5cee5471-dc6f-db16-8914-75ad5ad15...@m00nbsd.net>,
Maxime Villard   wrote:
>Le 04/10/2017 à 21:00, Maxime Villard a écrit :
>> Here is a Kernel ASLR implementation for NetBSD-amd64.
>> [...]
>> Known issues:
>>  * Right now, the kernel segments are contiguous. Starting from this
>>implementation, it wouldn't be really difficult to randomize the segments
>>independently - adding gaps between them and changing their order too.
>>Then, we could split the segments themselves in sub-blocks and intertwine
>>them.
>> [...]
>
>So, I did it. Now the kernel sections are split in sub-blocks, and are all
>randomized independently. See my drawing [1].
>
>What it means in practice, is that Kernel ASLR is much more difficult to
>defeat: a cache attack will at most allow you to know that a given range is
>mapped as executable for example, but you don't know which sub-block of .text
>it is; a kernel pointer leak will at most allow you to reconstruct the layout
>of one sub-block, but you don't know the layout and address of the remaining
>blocks, and there can be many.
>
>The size and number of these blocks is controlled by the split-by-file
>parameter in Makefile.amd64. Right now it is set to 2MB, which produces a
>kernel with ~23 allocatable (ie useful at runtime) sections, which is a third
>of the total number supported (BTSPACE_NSEGS = 64). I will probably reduce
>this parameter a bit in the future, to 1.5MB, or even 1MB.
>
>All of that leaves us with about the most advanced KASLR implementation
>available out there. There are ways to improve it even more, but you'll have
>to wait a few weeks for that.
>
>If you want to try it out you need to make sure you have the latest versions
>of GENERIC_KASLR / prekern / bootloader. The instructions are still here [2],
>and haven't changed.

Very nicely done!

christos

Re: amd64: kernel aslr support

[Maxime Villard]

Le 04/10/2017 à 21:00, Maxime Villard a écrit :

Here is a Kernel ASLR implementation for NetBSD-amd64.
[...]
Known issues:
 * Right now, the kernel segments are contiguous. Starting from this
   implementation, it wouldn't be really difficult to randomize the segments
   independently - adding gaps between them and changing their order too.
   Then, we could split the segments themselves in sub-blocks and intertwine
   them.
[...]


So, I did it. Now the kernel sections are split in sub-blocks, and are all
randomized independently. See my drawing [1].

What it means in practice, is that Kernel ASLR is much more difficult to
defeat: a cache attack will at most allow you to know that a given range is
mapped as executable for example, but you don't know which sub-block of .text
it is; a kernel pointer leak will at most allow you to reconstruct the layout
of one sub-block, but you don't know the layout and address of the remaining
blocks, and there can be many.

The size and number of these blocks is controlled by the split-by-file
parameter in Makefile.amd64. Right now it is set to 2MB, which produces a
kernel with ~23 allocatable (ie useful at runtime) sections, which is a third
of the total number supported (BTSPACE_NSEGS = 64). I will probably reduce
this parameter a bit in the future, to 1.5MB, or even 1MB.

All of that leaves us with about the most advanced KASLR implementation
available out there. There are ways to improve it even more, but you'll have
to wait a few weeks for that.

If you want to try it out you need to make sure you have the latest versions
of GENERIC_KASLR / prekern / bootloader. The instructions are still here [2],
and haven't changed.

Maxime

[1] http://mail-index.netbsd.org/source-changes/2017/11/14/msg089697.html
[2] http://m00nbsd.net/542a5cfd448aaf7db7adcadce74123d2.html

Re: amd64: kernel aslr support

[Maxime Villard]

Le 07/10/2017 à 21:29, Valery Ushakov a écrit :

On Sat, Oct 07, 2017 at 20:42:58 +0200, Maxime Villard wrote:


Le 04/10/2017 ? 21:00, Maxime Villard a ?crit :

Here is a Kernel ASLR implementation for NetBSD-amd64.
[...]
Known issues:
[...]
  * There are several redefinitions in the prekern headers. The way to remove
them depends on where we put the prekern in the source tree.


Does someone have a preference on where to put the prekern? I guess I'll
put it in src/sys/arch/amd64/prekern/.


I'd say src/sys/arch/amd64/stand/prekern to conform to existing
practice.


Alright, I did it. As I said in the commit message, it's not yet linked to
the build system, but basically it works.

You can build and install it this way:
cd /usr/src/sys/arch/amd64/stand/prekern
make
cp prekern /

Maxime

Re: amd64: kernel aslr support

[Valery Ushakov]

On Sat, Oct 07, 2017 at 20:42:58 +0200, Maxime Villard wrote:

> Le 04/10/2017 ? 21:00, Maxime Villard a ?crit :
> > Here is a Kernel ASLR implementation for NetBSD-amd64.
> > [...]
> > Known issues:
> > [...]
> >  * There are several redefinitions in the prekern headers. The way to remove
> >them depends on where we put the prekern in the source tree.
> 
> Does someone have a preference on where to put the prekern? I guess I'll
> put it in src/sys/arch/amd64/prekern/.

I'd say src/sys/arch/amd64/stand/prekern to conform to existing
practice.

-uwe

Re: amd64: kernel aslr support

[Maxime Villard]

Le 04/10/2017 à 21:00, Maxime Villard a écrit :

Here is a Kernel ASLR implementation for NetBSD-amd64.
[...]
Known issues:
[...]
 * There are several redefinitions in the prekern headers. The way to remove
   them depends on where we put the prekern in the source tree.


Does someone have a preference on where to put the prekern? I guess I'll
put it in src/sys/arch/amd64/prekern/.

Re: amd64: kernel aslr support

[Maxime Villard]

Le 06/10/2017 à 08:19, Martin Husemann a écrit :

On Thu, Oct 05, 2017 at 12:56:02PM +0200, Maxime Villard wrote:

I don't think it is possible to compile some parts as relocatable and some
others as static. What we could do is compile both the kernel and the prekern
separately, and use objcopy to merge them.


I don't see why this would be a problem.


I didn't say objcopy is a problem, the problem is the bootloader.

Re: amd64: kernel aslr support

[Mouse]

 Is live-kernel update more viable with this approach?
>>> Live kernel update is a much more complicated business, [...]
>> [I]t occurs to me that [the "live-kernel update" text] could have
>> been intended to refer not to updating the kernel without disturbing
>> a running system but rather to something more along the lines of
>> [making] the kernel [] play bootloader for a new kernel.
> I think the real point is updating a running system without
> interrupting the services it provides (apache etc).

Probably could be done, but it would need a lot more work than just
this.  As I understand the prekern, it would be of almost no use for
that; most of the pain will, I think, be in migrating a running
userland between kernels (especially if they are different in certain
important ways - if the kernel/user ABI differs, for example, I doubt
anything using the affected portions can migrate).

> There is, I think, little interest in pseudo-reboots via the prekern,
> except wanting a fast reboot - which is still not really useful.

It would be useful to me.  (Well, not in this form; I do not expect to
be using this prekern. I just mean I would really like to see reboots
cut out the time between the old kernel deciding to reboot and the
bootloader loading the new kernel.  The trip back through the BIOS can,
and on some of my machines does, turn what could be a fifteen- or
thirty-second reboot into a multi-minute reboot.)

/~\ The ASCII Mouse
\ / Ribbon Campaign
 X  Against HTMLmo...@rodents-montreal.org
/ \ Email!   7D C8 61 52 5D E7 2D 39  4E F1 31 3E E8 B3 27 4B

Re: amd64: kernel aslr support

[Maxime Villard]

Le 06/10/2017 à 10:37, Martin Husemann a écrit :

I don't get the issue with the bootloader - it just loads one binary
and doesn't care about ASLR, the "prekernel" entry point handles that
later (rearranging any mappings the bootloader might have previously
done afterwards)?


The bootloader loads *two* binaries: the prekern with loadfile_static(), and
the kernel with loadfile_dynamic().

The former is used by all the architectures we support, and it uses the memory
address referenced in the ELF segments. The latter chooses an address (see my
XXX in bootloader.diff), and maps the sections starting from there.

Basically, the problem is that if the prekern and the kernel are merged in one
binary, we're forced to have only one function, and it will be shared by all
architectures. Here, it becomes complicated to implement both the static and
dynamic loading, and it will certainly result in breakages or inconsistencies.

Beyond this bootloader issue, having dynamic+static sections in a
single binary is not structurally correct from an ELF pov.

Maxime

Re: amd64: kernel aslr support

[Martin Husemann]

On Fri, Oct 06, 2017 at 10:32:01AM +0200, Maxime Villard wrote:
> Le 06/10/2017 à 08:19, Martin Husemann a écrit :
> > On Thu, Oct 05, 2017 at 12:56:02PM +0200, Maxime Villard wrote:
> > > I don't think it is possible to compile some parts as relocatable and some
> > > others as static. What we could do is compile both the kernel and the 
> > > prekern
> > > separately, and use objcopy to merge them.
> > 
> > I don't see why this would be a problem.
> 
> I didn't say objcopy is a problem, the problem is the bootloader.

I don't even think objcopy would be needed, but there is always the
devil in the details and I didn't try.

I don't get the issue with the bootloader - it just loads one binary
and doesn't care about ASLR, the "prekernel" entry point handles that
later (rearranging any mappings the bootloader might have previously
done afterwards)?

Martin

Re: amd64: kernel aslr support

[Maxime Villard]

Le 06/10/2017 à 02:30, Thor Lancelot Simon a écrit :

  * The RNG is not really strong. Help in this area would be greatly
appreciated.


This is tricky mostly because once you start probing for hardware
devices or even CPU features, you're going to find yourself wanting
more and more of the support you'd get from the "real kernel".

For example, to probe for RDRAND support on the CPU, you need a
whole pile of CPU feature decoding.  To probe for environmental
sensors or an audio device you may need to know a whole pile about
ACPI and/or PCI.  And so forth.

EFI has a RNG API, but I think it's usually just stubbed out and
besides, you can't rely on having EFI...

I think I'd suggest some combination of:

* Just enough CPU-feature support to find/use RDRAND
  (Intel's sample code is not that big and I think it's
   suitably-licensed)

* Hash the contents of the "CMOS RAM" and/or EFI boot variables

* Maybe poke around for an IPMI BMC (has environmental sensors),
  or a TPM (has a RNG) on the LPC bus

* Maybe poke around for on-die temperature/voltage sensors
  (will again require some CPU identification support).

* Rather than just using rdtsc once, consider using rdtsc to
  "time" multiple hardware oscillators against one another;
  at the very least, you've always got the hardware clock.

* Also, you can use rdtsc to time memory accesses.

For quick and dirty "entropy extraction", you can crunch as much of this
data as you're able to connect together using SHA512.


As you said, all of this requires some heavy identification code. Besides, it
will be complicated on systems that don't have these features (I don't have
RDRAND for example).

Initially, I was more thinking about extending the rndsave_t structure with
fields dedicated exclusively to the prekern. The bootloader gives this
structure to the kernel for early entropy (BI_MODULE_RND), generated from the
previous run; the kernel could easily add a random uint32_t in it, which the
prekern uses right away.

Maxime

Re: amd64: kernel aslr support

[Maxime Villard]

Le 05/10/2017 à 17:39, Mouse a écrit :

Is live-kernel update more viable with this approach?

Live kernel update is a much more complicated business, [...]


I didn't write that the bit about live-kernel updates, but it occurs to
me that it could have been intended to refer not to updating the kernel
without disturbing a running system but rather to something more along
the lines of...kexec(), I think it was?  Basically, make the kernel -
or, here, the prekern - play bootloader for a new kernel.  Reboot in
most senses without actually dropping back to the hardware's BIOS (or
moral equivalent, for non-peecees).


I think the real point is updating a running system without interrupting the
services it provides (apache etc). There is, I think, little interest in
pseudo-reboots via the prekern, except wanting a fast reboot - which is still
not really useful.

Maxime

Re: amd64: kernel aslr support

[Martin Husemann]

On Thu, Oct 05, 2017 at 12:56:02PM +0200, Maxime Villard wrote:
> I don't think it is possible to compile some parts as relocatable and some
> others as static. What we could do is compile both the kernel and the prekern
> separately, and use objcopy to merge them.

I don't see why this would be a problem.

Martin

Re: amd64: kernel aslr support

[John Nemeth]

On Oct 5,  8:30pm, Thor Lancelot Simon wrote:
}
} >  * The RNG is not really strong. Help in this area would be greatly
} >appreciated.
} 
} This is tricky mostly because once you start probing for hardware
} devices or even CPU features, you're going to find yourself wanting
} more and more of the support you'd get from the "real kernel".
} 
} For example, to probe for RDRAND support on the CPU, you need a
} whole pile of CPU feature decoding.  To probe for environmental
} sensors or an audio device you may need to know a whole pile about
} ACPI and/or PCI.  And so forth.
} 
} EFI has a RNG API, but I think it's usually just stubbed out and
} besides, you can't rely on having EFI...

 It does, but it isn't listed as a runtime service, so likely
isn't available after ExitBootServices() is called, which would be
called by efiboot.

} I think I'd suggest some combination of:
} 
}   * Just enough CPU-feature support to find/use RDRAND
} (Intel's sample code is not that big and I think it's
}  suitably-licensed)
} 
}   * Hash the contents of the "CMOS RAM" and/or EFI boot variables

 We currently have no support, of which I'm aware, for accessing
EFI bot variables.

}   * Maybe poke around for an IPMI BMC (has environmental sensors),
} or a TPM (has a RNG) on the LPC bus
} 
}   * Maybe poke around for on-die temperature/voltage sensors
} (will again require some CPU identification support).
} 
}   * Rather than just using rdtsc once, consider using rdtsc to
} "time" multiple hardware oscillators against one another;
} at the very least, you've always got the hardware clock.
} 
}   * Also, you can use rdtsc to time memory accesses.
} 
} For quick and dirty "entropy extraction", you can crunch as much of this
} data as you're able to connect together using SHA512.
} 
} I know, little or none of this is easy.
} 
} Thor
}-- End of excerpt from Thor Lancelot Simon

Re: amd64: kernel aslr support

[Thor Lancelot Simon]

>  * The RNG is not really strong. Help in this area would be greatly
>appreciated.

This is tricky mostly because once you start probing for hardware
devices or even CPU features, you're going to find yourself wanting
more and more of the support you'd get from the "real kernel".

For example, to probe for RDRAND support on the CPU, you need a
whole pile of CPU feature decoding.  To probe for environmental
sensors or an audio device you may need to know a whole pile about
ACPI and/or PCI.  And so forth.

EFI has a RNG API, but I think it's usually just stubbed out and
besides, you can't rely on having EFI...

I think I'd suggest some combination of:

* Just enough CPU-feature support to find/use RDRAND
  (Intel's sample code is not that big and I think it's
   suitably-licensed)

* Hash the contents of the "CMOS RAM" and/or EFI boot variables

* Maybe poke around for an IPMI BMC (has environmental sensors),
  or a TPM (has a RNG) on the LPC bus

* Maybe poke around for on-die temperature/voltage sensors
  (will again require some CPU identification support).

* Rather than just using rdtsc once, consider using rdtsc to
  "time" multiple hardware oscillators against one another;
  at the very least, you've always got the hardware clock.

* Also, you can use rdtsc to time memory accesses.

For quick and dirty "entropy extraction", you can crunch as much of this
data as you're able to connect together using SHA512.

I know, little or none of this is easy.

Thor

Re: amd64: kernel aslr support

[Mouse]

>> Is live-kernel update more viable with this approach?
> Live kernel update is a much more complicated business, [...]

I didn't write that the bit about live-kernel updates, but it occurs to
me that it could have been intended to refer not to updating the kernel
without disturbing a running system but rather to something more along
the lines of...kexec(), I think it was?  Basically, make the kernel -
or, here, the prekern - play bootloader for a new kernel.  Reboot in
most senses without actually dropping back to the hardware's BIOS (or
moral equivalent, for non-peecees).

/~\ The ASCII Mouse
\ / Ribbon Campaign
 X  Against HTMLmo...@rodents-montreal.org
/ \ Email!   7D C8 61 52 5D E7 2D 39  4E F1 31 3E E8 B3 27 4B

Re: amd64: kernel aslr support

[Maxime Villard]

Le 05/10/2017 à 14:59, Kamil Rytarowski a écrit :

On 05.10.2017 12:56, Maxime Villard wrote:


The advantage of having a separate prekern is that you can update the
kernel
without touching the prekern, or the other way around.


Is live-kernel update more viable with this approach?


It's not what I had in mind when saying "update"; I just meant that since
the components are in different files, you can change one without changing
the other.

Live kernel update is a much more complicated business, but as far as I can
tell, whether the prekern is separate from the kernel or not does not
essentially change anything.

Maxime

Re: amd64: kernel aslr support

[Kamil Rytarowski]

On 05.10.2017 12:56, Maxime Villard wrote:
> 
> The advantage of having a separate prekern is that you can update the
> kernel
> without touching the prekern, or the other way around.

Is live-kernel update more viable with this approach?



signature.asc
Description: OpenPGP digital signature

Re: amd64: kernel aslr support

[Maxime Villard]

Le 05/10/2017 à 09:12, Martin Husemann a écrit :

On Wed, Oct 04, 2017 at 09:00:50PM +0200, Maxime Villard wrote:

This implementation is based on a specialized kernel, called the prekern, which
relocates the real NetBSD kernel. The bootloader loads both the prekern and the
kernel in memory, and jumps into the prekern with a set of arguments; the
prekern relocates the kernel and then jumps into it, passing it the arguments
from the bootloader plus several other values.


Can you explain why this needs a specialized prekern?
Can't you just use the normal kernel binary and a make all of prekern
a special function inside the main kernel, maybe in a separate section
that is then not reloacted?


I don't think it is possible to compile some parts as relocatable and some
others as static. What we could do is compile both the kernel and the prekern
separately, and use objcopy to merge them.

But the real issue is the bootloader. If you look for, you can see that my
bootloader patch does not change the loadfile_static function - which is used
by all the architectures. Modifying it to support kernels that are both dynamic
and static is complicated and adds to the already existing mess, and we can
expect breakages on some obscure architectures.

(as a side note, for the next GSOC there really should be a "cleanup libsa and
all our bootloaders" project...)

The advantage of having a separate prekern is that you can update the kernel
without touching the prekern, or the other way around.

Maxime

Re: amd64: kernel aslr support

[Manuel Bouyer]

On Wed, Oct 04, 2017 at 06:59:02PM -0400, Mouse wrote:
> > Here is a Kernel ASLR implementation for NetBSD-amd64.  [...]
> 
> > Contrary to what has been said in previous discussions, KASLR does
> > not alter debugability in any way: the symbols are still mapped in
> > memory as they are right now, and ddb resolves them as usual.
> 
> I disagree that that's enough to "not alter debugability in any way".
> It makes common debugging tasks work, perhaps, but I have occasionally
> run into cases where all I have is a pc value (even less often, but
> it's still happened, a data address) and have had nothing but /netbsd's
> symbol table to help me make sense of it.

Sure. One common case is Xen.

It may also alter reproductablility, in case of pointer bug in the kernel.

I think the base address of each section should be printed to console at boot,
so that with a serial console log we can make sense of the addresses.

-- 
Manuel Bouyer 
 NetBSD: 26 ans d'experience feront toujours la difference
--

Re: amd64: kernel aslr support

[Kamil Rytarowski]

On 04.10.2017 21:00, Maxime Villard wrote:
> Here is a Kernel ASLR implementation for NetBSD-amd64. It is light,
> functional,
> user-friendly, and does not break any feature of the system.
> Instructions on
> how to install and use it can be found here [1].

I'm looking forward to see it aboard in mainline.

I'm also asking why it needs a separate prekern, and not being
integrated in the kernel?

How much is it MI-friendly?

This is an enterprise standard feature today. I noted that Fuchsia
developers enabled kaslr even before finishing their scheduler.

In terms of kasan (and certainly ktsan) + kaslr coexistence, we need to
reserve a shadow buffer that is 1/8 of the allocated kernel memory. So
something like 1/7 of potential memory that is contiguous should be
untouched to make kasan easier.



signature.asc
Description: OpenPGP digital signature

Re: amd64: kernel aslr support

[Martin Husemann]

On Wed, Oct 04, 2017 at 09:00:50PM +0200, Maxime Villard wrote:
> This implementation is based on a specialized kernel, called the prekern, 
> which
> relocates the real NetBSD kernel. The bootloader loads both the prekern and 
> the
> kernel in memory, and jumps into the prekern with a set of arguments; the
> prekern relocates the kernel and then jumps into it, passing it the arguments
> from the bootloader plus several other values.

Can you explain why this needs a specialized prekern?
Can't you just use the normal kernel binary and a make all of prekern
a special function inside the main kernel, maybe in a separate section
that is then not reloacted?

Martin

Re: amd64: kernel aslr support

[Maxime Villard]

Le 05/10/2017 à 00:59, Mouse a écrit :

Here is a Kernel ASLR implementation for NetBSD-amd64.  [...]



Contrary to what has been said in previous discussions, KASLR does
not alter debugability in any way: the symbols are still mapped in
memory as they are right now, and ddb resolves them as usual.


I disagree that that's enough to "not alter debugability in any way".
It makes common debugging tasks work, perhaps, but I have occasionally
run into cases where all I have is a pc value (even less often, but
it's still happened, a data address) and have had nothing but /netbsd's
symbol table to help me make sense of it.


What I meant to say, is that the information ddb usually gives you is not
altered as a result of running KASLR; but you are right to say that it is
more complicated for you to get information by yourself manually.

(thinking about it, I could add the sections addresses in the ddb output,
and add a command to print %rip or whatever else relatively to the section
it finds itself in)

Maxime

Re: amd64: kernel aslr support

[David Holland]

On Wed, Oct 04, 2017 at 06:59:02PM -0400, Mouse wrote:
 > > Here is a Kernel ASLR implementation for NetBSD-amd64.  [...]
 > 
 > > Contrary to what has been said in previous discussions, KASLR does
 > > not alter debugability in any way: the symbols are still mapped in
 > > memory as they are right now, and ddb resolves them as usual.
 > 
 > I disagree that that's enough to "not alter debugability in any way".
 > It makes common debugging tasks work, perhaps, but I have occasionally
 > run into cases where all I have is a pc value (even less often, but
 > it's still happened, a data address) and have had nothing but /netbsd's
 > symbol table to help me make sense of it.

Often also this is all one gets in a bug report. Not infrequently
these come from people who aren't really prepared to do much more to
investigate themselves than run nm -n /netbsd when asked.

-- 
David A. Holland
dholl...@netbsd.org

Re: amd64: kernel aslr support

[Mouse]

> Here is a Kernel ASLR implementation for NetBSD-amd64.  [...]

> Contrary to what has been said in previous discussions, KASLR does
> not alter debugability in any way: the symbols are still mapped in
> memory as they are right now, and ddb resolves them as usual.

I disagree that that's enough to "not alter debugability in any way".
It makes common debugging tasks work, perhaps, but I have occasionally
run into cases where all I have is a pc value (even less often, but
it's still happened, a data address) and have had nothing but /netbsd's
symbol table to help me make sense of it.

/~\ The ASCII Mouse
\ / Ribbon Campaign
 X  Against HTMLmo...@rodents-montreal.org
/ \ Email!   7D C8 61 52 5D E7 2D 39  4E F1 31 3E E8 B3 27 4B

amd64: kernel aslr support

[Maxime Villard]

Here is a Kernel ASLR implementation for NetBSD-amd64. It is light, functional,
user-friendly, and does not break any feature of the system. Instructions on
how to install and use it can be found here [1].

There are three patches, incompatible.diff is the one that needs more work or
proper #ifdefs. I give here a brief summary, but basically if you want to
understand how the implementation works in details, you need to read the code.

This implementation is based on a specialized kernel, called the prekern, which
relocates the real NetBSD kernel. The bootloader loads both the prekern and the
kernel in memory, and jumps into the prekern with a set of arguments; the
prekern relocates the kernel and then jumps into it, passing it the arguments
from the bootloader plus several other values.

The kernel is entered in the same state as the one created by the generic
locore: 64bit mode, paging, and several cpu features enabled. The only cleanup
it has to do is unmapping the prekern. The physical space of the prekern gets
recycled by UVM.

Before the kernel initializes a new IDT, it lives with that of the prekern.
Therefore, if the kernel faults early - assuming due to a mistake in the passed
arguments -, the prekern is re-entered and the exception is handled safely.

Known issues:
 * Right now, the kernel segments are contiguous. Starting from this
   implementation, it wouldn't be really difficult to randomize the segments
   independently - adding gaps between them and changing their order too.
   Then, we could split the segments themselves in sub-blocks and intertwine
   them.
 * In the set of patches, I'm sometimes replacing PA+KERNBASE by a pointer to
   the direct map. That's because I haven't found a clean way of doing it with
   bootspace yet - it will be solved later.
 * The kernel is not mapped with large pages. Doing so depends on the alignment
   the bootloader uses (KERNALIGN). This too will be solved later.
 * The RNG is not really strong. Help in this area would be greatly
   appreciated.
 * There are several redefinitions in the prekern headers. The way to remove
   them depends on where we put the prekern in the source tree.
 * And several other minor things that will be fixed soon...

Once the RNG is more robust, it is not complicated to randomize the kernel
heap too, and end up with strictly all of the kernel VA randomized.

Contrary to what has been said in previous discussions, KASLR does not alter
debugability in any way: the symbols are still mapped in memory as they are
right now, and ddb resolves them as usual.

Regarding address exposure, several entry points will have to be changed not
to disclose kernel VAs (eg "modstat -k"), but that's another debate. Finally,
when it comes to the performance impact, I don't have a very clear answer; we
don't use large pages yet so it's necessarily slower than usual. This being
said, I didn't experience any extravagant performance regression.

Comments, suggestions, tests and feedbacks are welcome.

Maxime

[1] http://m00nbsd.net/542a5cfd448aaf7db7adcadce74123d2.html