Re: [PATCH 3/4] x86/boot/compressed/64: Introduce place_trampoline()

[Ingo Molnar]

* Kirill A. Shutemov  wrote:

> > One other detail I noticed:
> > 
> > /* Bound size of trampoline code */
> > .orglvl5_trampoline_src + LVL5_TRAMPOLINE_CODE_SIZE
> > 
> > will this generate a build error if the trampoline code exceeds 0x40?
> 
> Yes, this is the point. Just a failsafe if trampoline code would grew too
> much.

Ok, good!

Thanks,

Ingo

Re: [PATCH 3/4] x86/boot/compressed/64: Introduce place_trampoline()

[Ingo Molnar]

* Kirill A. Shutemov  wrote:

> > One other detail I noticed:
> > 
> > /* Bound size of trampoline code */
> > .orglvl5_trampoline_src + LVL5_TRAMPOLINE_CODE_SIZE
> > 
> > will this generate a build error if the trampoline code exceeds 0x40?
> 
> Yes, this is the point. Just a failsafe if trampoline code would grew too
> much.

Ok, good!

Thanks,

Ingo

Re: [PATCH 3/4] x86/boot/compressed/64: Introduce place_trampoline()

[Kirill A. Shutemov]

On Fri, Nov 10, 2017 at 10:29:33AM +0100, Ingo Molnar wrote:
> 
> * Kirill A. Shutemov  wrote:
> 
> > --- a/arch/x86/boot/compressed/head_64.S
> > +++ b/arch/x86/boot/compressed/head_64.S
> > @@ -315,6 +315,18 @@ ENTRY(startup_64)
> >  * The first step is go into compatibility mode.
> >  */
> >  
> > +   /*
> > +* Find suitable place for trampoline and populate it.
> > +* The address will be stored in RCX.
> > +*
> > +* RSI holds real mode data and need to be preserved across
> > +* a function call.
> > +*/
> > +   pushq   %rsi
> > +   callplace_trampoline
> > +   popq%rsi
> > +   movq%rax, %rcx
> > +
> > /* Clear additional page table */
> > leaqlvl5_pgtable(%rbx), %rdi
> > xorq%rax, %rax
> 
> So in the final version of this code we now have:
> 
>   pushq   %rsi
>   callneed_to_enabled_l5
>   popq%rsi
> 
>   /* If need_to_enabled_l5() returned zero, we're done here. */
>   cmpq$0, %rax
>   je  lvl5
> 
>   /*
>* At this point we are in long mode with 4-level paging enabled,
>* but we want to enable 5-level paging.
>*
>* The problem is that we cannot do it directly. Setting LA57 in
>* long mode would trigger #GP. So we need to switch off long mode
>* first.
>*
>* We use trampoline in lower memory to handle situation when
>* bootloader put the kernel image above 4G.
>*
>* The first step is go into compatibility mode.
>*/
> 
>   /*
>* Find suitable place for trampoline and populate it.
>* The address will be stored in RCX.
>*
>* RSI holds real mode data and need to be preserved across
>* a function call.
>*/
>   pushq   %rsi
>   callplace_trampoline
>   popq%rsi
>   movq%rax, %rcx
> 
> Firstly, the 'need_to_enabled_l5' name sucks because it includes a typo, but 
> also 
> because the prefix is way too generic.
> 
> Something like:
> 
>   l5_paging_required()
> 
> would read a lot better - and would also provide a namespace for all L5 
> paging 
> related functions.
> 
> Secondly, couldn't this be combined into a single .c function, named 
> accordingly:
> 
>   l5_paging_prepare()
> 
> which would return true if L5 paging is available and should be enabled. In 
> this 
> case the trampoline copying function would be called in C, by 
> l5_paging_prepare().
> 
> This further reduces the amount of assembly code.

Makes sense.

-- 
 Kirill A. Shutemov

Re: [PATCH 3/4] x86/boot/compressed/64: Introduce place_trampoline()

[Kirill A. Shutemov]

On Fri, Nov 10, 2017 at 10:29:33AM +0100, Ingo Molnar wrote:
> 
> * Kirill A. Shutemov  wrote:
> 
> > --- a/arch/x86/boot/compressed/head_64.S
> > +++ b/arch/x86/boot/compressed/head_64.S
> > @@ -315,6 +315,18 @@ ENTRY(startup_64)
> >  * The first step is go into compatibility mode.
> >  */
> >  
> > +   /*
> > +* Find suitable place for trampoline and populate it.
> > +* The address will be stored in RCX.
> > +*
> > +* RSI holds real mode data and need to be preserved across
> > +* a function call.
> > +*/
> > +   pushq   %rsi
> > +   callplace_trampoline
> > +   popq%rsi
> > +   movq%rax, %rcx
> > +
> > /* Clear additional page table */
> > leaqlvl5_pgtable(%rbx), %rdi
> > xorq%rax, %rax
> 
> So in the final version of this code we now have:
> 
>   pushq   %rsi
>   callneed_to_enabled_l5
>   popq%rsi
> 
>   /* If need_to_enabled_l5() returned zero, we're done here. */
>   cmpq$0, %rax
>   je  lvl5
> 
>   /*
>* At this point we are in long mode with 4-level paging enabled,
>* but we want to enable 5-level paging.
>*
>* The problem is that we cannot do it directly. Setting LA57 in
>* long mode would trigger #GP. So we need to switch off long mode
>* first.
>*
>* We use trampoline in lower memory to handle situation when
>* bootloader put the kernel image above 4G.
>*
>* The first step is go into compatibility mode.
>*/
> 
>   /*
>* Find suitable place for trampoline and populate it.
>* The address will be stored in RCX.
>*
>* RSI holds real mode data and need to be preserved across
>* a function call.
>*/
>   pushq   %rsi
>   callplace_trampoline
>   popq%rsi
>   movq%rax, %rcx
> 
> Firstly, the 'need_to_enabled_l5' name sucks because it includes a typo, but 
> also 
> because the prefix is way too generic.
> 
> Something like:
> 
>   l5_paging_required()
> 
> would read a lot better - and would also provide a namespace for all L5 
> paging 
> related functions.
> 
> Secondly, couldn't this be combined into a single .c function, named 
> accordingly:
> 
>   l5_paging_prepare()
> 
> which would return true if L5 paging is available and should be enabled. In 
> this 
> case the trampoline copying function would be called in C, by 
> l5_paging_prepare().
> 
> This further reduces the amount of assembly code.

Makes sense.

-- 
 Kirill A. Shutemov

Re: [PATCH 3/4] x86/boot/compressed/64: Introduce place_trampoline()

[Kirill A. Shutemov]

On Fri, Nov 10, 2017 at 10:28:12AM +0100, Ingo Molnar wrote:
> 
> * Ingo Molnar  wrote:
> 
> > > --- a/arch/x86/boot/compressed/head_64.S
> > > +++ b/arch/x86/boot/compressed/head_64.S
> > > @@ -315,6 +315,18 @@ ENTRY(startup_64)
> > >* The first step is go into compatibility mode.
> > >*/
> > >  
> > > + /*
> > > +  * Find suitable place for trampoline and populate it.
> > > +  * The address will be stored in RCX.
> > > +  *
> > > +  * RSI holds real mode data and need to be preserved across
> > > +  * a function call.
> > > +  */
> > > + pushq   %rsi
> > > + callplace_trampoline
> > > + popq%rsi
> > > + movq%rax, %rcx
> > > +
> > >   /* Clear additional page table */
> > >   leaqlvl5_pgtable(%rbx), %rdi
> > >   xorq%rax, %rax
> > 
> > One request: it's always going to be fragile if the _only_ thing that uses 
> > the 
> > trampoline is the 5-level paging code.
> > 
> > Could we use the trampoline in the 4-level paging case too? It's not 
> > required, but 
> > would test much of the trampoline allocation and copying machinery - and 
> > the 
> > performance cost is negligible.
> 
> Note that right now the trampoline is pointless on 4-level setups, so there's 
> nothing to copy - but we could perhaps make it meaningful. But maybe it's not 
> a 
> good idea.

Let me see how it will play out.

> One other detail I noticed:
> 
> /* Bound size of trampoline code */
> .orglvl5_trampoline_src + LVL5_TRAMPOLINE_CODE_SIZE
> 
> will this generate a build error if the trampoline code exceeds 0x40?

Yes, this is the point. Just a failsafe if trampoline code would grew too
much.

-- 
 Kirill A. Shutemov

Re: [PATCH 3/4] x86/boot/compressed/64: Introduce place_trampoline()

[Kirill A. Shutemov]

On Fri, Nov 10, 2017 at 10:28:12AM +0100, Ingo Molnar wrote:
> 
> * Ingo Molnar  wrote:
> 
> > > --- a/arch/x86/boot/compressed/head_64.S
> > > +++ b/arch/x86/boot/compressed/head_64.S
> > > @@ -315,6 +315,18 @@ ENTRY(startup_64)
> > >* The first step is go into compatibility mode.
> > >*/
> > >  
> > > + /*
> > > +  * Find suitable place for trampoline and populate it.
> > > +  * The address will be stored in RCX.
> > > +  *
> > > +  * RSI holds real mode data and need to be preserved across
> > > +  * a function call.
> > > +  */
> > > + pushq   %rsi
> > > + callplace_trampoline
> > > + popq%rsi
> > > + movq%rax, %rcx
> > > +
> > >   /* Clear additional page table */
> > >   leaqlvl5_pgtable(%rbx), %rdi
> > >   xorq%rax, %rax
> > 
> > One request: it's always going to be fragile if the _only_ thing that uses 
> > the 
> > trampoline is the 5-level paging code.
> > 
> > Could we use the trampoline in the 4-level paging case too? It's not 
> > required, but 
> > would test much of the trampoline allocation and copying machinery - and 
> > the 
> > performance cost is negligible.
> 
> Note that right now the trampoline is pointless on 4-level setups, so there's 
> nothing to copy - but we could perhaps make it meaningful. But maybe it's not 
> a 
> good idea.

Let me see how it will play out.

> One other detail I noticed:
> 
> /* Bound size of trampoline code */
> .orglvl5_trampoline_src + LVL5_TRAMPOLINE_CODE_SIZE
> 
> will this generate a build error if the trampoline code exceeds 0x40?

Yes, this is the point. Just a failsafe if trampoline code would grew too
much.

-- 
 Kirill A. Shutemov

Re: [PATCH 3/4] x86/boot/compressed/64: Introduce place_trampoline()

[Kirill A. Shutemov]

On Fri, Nov 10, 2017 at 10:14:53AM +0100, Ingo Molnar wrote:
> I'm wondering, how did you test it if no current bootloader puts the kernel 
> to 
> above addresses of 4G?

I didn't test it directly.

I've checked with debugger that there's no memory accesses above 1M
between disabling paging in the trampoline and returning back from
trampoline to the main kernel image code.

-- 
 Kirill A. Shutemov

Re: [PATCH 3/4] x86/boot/compressed/64: Introduce place_trampoline()

[Kirill A. Shutemov]

On Fri, Nov 10, 2017 at 10:14:53AM +0100, Ingo Molnar wrote:
> I'm wondering, how did you test it if no current bootloader puts the kernel 
> to 
> above addresses of 4G?

I didn't test it directly.

I've checked with debugger that there's no memory accesses above 1M
between disabling paging in the trampoline and returning back from
trampoline to the main kernel image code.

-- 
 Kirill A. Shutemov

Re: [PATCH 3/4] x86/boot/compressed/64: Introduce place_trampoline()

[Ingo Molnar]

* Kirill A. Shutemov  wrote:

> --- a/arch/x86/boot/compressed/head_64.S
> +++ b/arch/x86/boot/compressed/head_64.S
> @@ -315,6 +315,18 @@ ENTRY(startup_64)
>* The first step is go into compatibility mode.
>*/
>  
> + /*
> +  * Find suitable place for trampoline and populate it.
> +  * The address will be stored in RCX.
> +  *
> +  * RSI holds real mode data and need to be preserved across
> +  * a function call.
> +  */
> + pushq   %rsi
> + callplace_trampoline
> + popq%rsi
> + movq%rax, %rcx
> +
>   /* Clear additional page table */
>   leaqlvl5_pgtable(%rbx), %rdi
>   xorq%rax, %rax

So in the final version of this code we now have:

pushq   %rsi
callneed_to_enabled_l5
popq%rsi

/* If need_to_enabled_l5() returned zero, we're done here. */
cmpq$0, %rax
je  lvl5

/*
 * At this point we are in long mode with 4-level paging enabled,
 * but we want to enable 5-level paging.
 *
 * The problem is that we cannot do it directly. Setting LA57 in
 * long mode would trigger #GP. So we need to switch off long mode
 * first.
 *
 * We use trampoline in lower memory to handle situation when
 * bootloader put the kernel image above 4G.
 *
 * The first step is go into compatibility mode.
 */

/*
 * Find suitable place for trampoline and populate it.
 * The address will be stored in RCX.
 *
 * RSI holds real mode data and need to be preserved across
 * a function call.
 */
pushq   %rsi
callplace_trampoline
popq%rsi
movq%rax, %rcx

Firstly, the 'need_to_enabled_l5' name sucks because it includes a typo, but 
also 
because the prefix is way too generic.

Something like:

l5_paging_required()

would read a lot better - and would also provide a namespace for all L5 paging 
related functions.

Secondly, couldn't this be combined into a single .c function, named 
accordingly:

l5_paging_prepare()

which would return true if L5 paging is available and should be enabled. In 
this 
case the trampoline copying function would be called in C, by 
l5_paging_prepare().

This further reduces the amount of assembly code.

Thanks,

Ingo

Re: [PATCH 3/4] x86/boot/compressed/64: Introduce place_trampoline()

[Ingo Molnar]

* Kirill A. Shutemov  wrote:

> --- a/arch/x86/boot/compressed/head_64.S
> +++ b/arch/x86/boot/compressed/head_64.S
> @@ -315,6 +315,18 @@ ENTRY(startup_64)
>* The first step is go into compatibility mode.
>*/
>  
> + /*
> +  * Find suitable place for trampoline and populate it.
> +  * The address will be stored in RCX.
> +  *
> +  * RSI holds real mode data and need to be preserved across
> +  * a function call.
> +  */
> + pushq   %rsi
> + callplace_trampoline
> + popq%rsi
> + movq%rax, %rcx
> +
>   /* Clear additional page table */
>   leaqlvl5_pgtable(%rbx), %rdi
>   xorq%rax, %rax

So in the final version of this code we now have:

pushq   %rsi
callneed_to_enabled_l5
popq%rsi

/* If need_to_enabled_l5() returned zero, we're done here. */
cmpq$0, %rax
je  lvl5

/*
 * At this point we are in long mode with 4-level paging enabled,
 * but we want to enable 5-level paging.
 *
 * The problem is that we cannot do it directly. Setting LA57 in
 * long mode would trigger #GP. So we need to switch off long mode
 * first.
 *
 * We use trampoline in lower memory to handle situation when
 * bootloader put the kernel image above 4G.
 *
 * The first step is go into compatibility mode.
 */

/*
 * Find suitable place for trampoline and populate it.
 * The address will be stored in RCX.
 *
 * RSI holds real mode data and need to be preserved across
 * a function call.
 */
pushq   %rsi
callplace_trampoline
popq%rsi
movq%rax, %rcx

Firstly, the 'need_to_enabled_l5' name sucks because it includes a typo, but 
also 
because the prefix is way too generic.

Something like:

l5_paging_required()

would read a lot better - and would also provide a namespace for all L5 paging 
related functions.

Secondly, couldn't this be combined into a single .c function, named 
accordingly:

l5_paging_prepare()

which would return true if L5 paging is available and should be enabled. In 
this 
case the trampoline copying function would be called in C, by 
l5_paging_prepare().

This further reduces the amount of assembly code.

Thanks,

Ingo

Re: [PATCH 3/4] x86/boot/compressed/64: Introduce place_trampoline()

[Ingo Molnar]

* Ingo Molnar  wrote:

> > --- a/arch/x86/boot/compressed/head_64.S
> > +++ b/arch/x86/boot/compressed/head_64.S
> > @@ -315,6 +315,18 @@ ENTRY(startup_64)
> >  * The first step is go into compatibility mode.
> >  */
> >  
> > +   /*
> > +* Find suitable place for trampoline and populate it.
> > +* The address will be stored in RCX.
> > +*
> > +* RSI holds real mode data and need to be preserved across
> > +* a function call.
> > +*/
> > +   pushq   %rsi
> > +   callplace_trampoline
> > +   popq%rsi
> > +   movq%rax, %rcx
> > +
> > /* Clear additional page table */
> > leaqlvl5_pgtable(%rbx), %rdi
> > xorq%rax, %rax
> 
> One request: it's always going to be fragile if the _only_ thing that uses 
> the 
> trampoline is the 5-level paging code.
> 
> Could we use the trampoline in the 4-level paging case too? It's not 
> required, but 
> would test much of the trampoline allocation and copying machinery - and the 
> performance cost is negligible.

Note that right now the trampoline is pointless on 4-level setups, so there's 
nothing to copy - but we could perhaps make it meaningful. But maybe it's not a 
good idea.

One other detail I noticed:

/* Bound size of trampoline code */
.orglvl5_trampoline_src + LVL5_TRAMPOLINE_CODE_SIZE

will this generate a build error if the trampoline code exceeds 0x40?

Thanks,

Ingo

Re: [PATCH 3/4] x86/boot/compressed/64: Introduce place_trampoline()

[Ingo Molnar]

* Ingo Molnar  wrote:

> > --- a/arch/x86/boot/compressed/head_64.S
> > +++ b/arch/x86/boot/compressed/head_64.S
> > @@ -315,6 +315,18 @@ ENTRY(startup_64)
> >  * The first step is go into compatibility mode.
> >  */
> >  
> > +   /*
> > +* Find suitable place for trampoline and populate it.
> > +* The address will be stored in RCX.
> > +*
> > +* RSI holds real mode data and need to be preserved across
> > +* a function call.
> > +*/
> > +   pushq   %rsi
> > +   callplace_trampoline
> > +   popq%rsi
> > +   movq%rax, %rcx
> > +
> > /* Clear additional page table */
> > leaqlvl5_pgtable(%rbx), %rdi
> > xorq%rax, %rax
> 
> One request: it's always going to be fragile if the _only_ thing that uses 
> the 
> trampoline is the 5-level paging code.
> 
> Could we use the trampoline in the 4-level paging case too? It's not 
> required, but 
> would test much of the trampoline allocation and copying machinery - and the 
> performance cost is negligible.

Note that right now the trampoline is pointless on 4-level setups, so there's 
nothing to copy - but we could perhaps make it meaningful. But maybe it's not a 
good idea.

One other detail I noticed:

/* Bound size of trampoline code */
.orglvl5_trampoline_src + LVL5_TRAMPOLINE_CODE_SIZE

will this generate a build error if the trampoline code exceeds 0x40?

Thanks,

Ingo

Re: [PATCH 3/4] x86/boot/compressed/64: Introduce place_trampoline()

[Ingo Molnar]

* Kirill A. Shutemov  wrote:

> If bootloader enables 64-bit mode with 4-level paging, we need to
> switch over to 5-level paging. The switching requires disabling paging.
> It works fine if kernel itself is loaded below 4G.
> 
> If bootloader put the kernel above 4G (not sure if anybody does this),
> we would loose control as soon as paging is disabled as code becomes
> unreachable.
> 
> To handle the situation, we need a trampoline in lower memory that would
> take care about switching on 5-level paging.
> 
> Apart from trampoline itself we also need place to store top level page
> table in lower memory as we don't have a way to load 64-bit value into
> CR3 from 32-bit mode. We only really need 8-bytes there as we only use
> the very first entry of the page table. But we allocate whole page
> anyway. We cannot have the code in the same because, there's hazard that
> a CPU would read page table speculatively and get confused seeing
> garbage.
> 
> This patch introduces place_trampoline() that finds right spot in lower
> memory for trampoline, copies trampoline code there and setups new top
> level page table for 5-level paging.
> 
> At this point we do all the preparation, but not yet use trampoline.
> It will be done in following patch.
> 
> Signed-off-by: Kirill A. Shutemov 
> ---
>  arch/x86/boot/compressed/head_64.S   | 13 +++
>  arch/x86/boot/compressed/pagetable.c | 42 
> 
>  arch/x86/boot/compressed/pagetable.h | 18 
>  3 files changed, 73 insertions(+)
>  create mode 100644 arch/x86/boot/compressed/pagetable.h
> 
> diff --git a/arch/x86/boot/compressed/head_64.S 
> b/arch/x86/boot/compressed/head_64.S
> index 6ac8239af2b6..4d1555b39de0 100644
> --- a/arch/x86/boot/compressed/head_64.S
> +++ b/arch/x86/boot/compressed/head_64.S
> @@ -315,6 +315,18 @@ ENTRY(startup_64)
>* The first step is go into compatibility mode.
>*/
>  
> + /*
> +  * Find suitable place for trampoline and populate it.
> +  * The address will be stored in RCX.
> +  *
> +  * RSI holds real mode data and need to be preserved across
> +  * a function call.
> +  */
> + pushq   %rsi
> + callplace_trampoline
> + popq%rsi
> + movq%rax, %rcx
> +
>   /* Clear additional page table */
>   leaqlvl5_pgtable(%rbx), %rdi
>   xorq%rax, %rax

One request: it's always going to be fragile if the _only_ thing that uses the 
trampoline is the 5-level paging code.

Could we use the trampoline in the 4-level paging case too? It's not required, 
but 
would test much of the trampoline allocation and copying machinery - and the 
performance cost is negligible.

Thanks,

Ingo

Re: [PATCH 3/4] x86/boot/compressed/64: Introduce place_trampoline()

[Ingo Molnar]

* Kirill A. Shutemov  wrote:

> If bootloader enables 64-bit mode with 4-level paging, we need to
> switch over to 5-level paging. The switching requires disabling paging.
> It works fine if kernel itself is loaded below 4G.
> 
> If bootloader put the kernel above 4G (not sure if anybody does this),
> we would loose control as soon as paging is disabled as code becomes
> unreachable.
> 
> To handle the situation, we need a trampoline in lower memory that would
> take care about switching on 5-level paging.
> 
> Apart from trampoline itself we also need place to store top level page
> table in lower memory as we don't have a way to load 64-bit value into
> CR3 from 32-bit mode. We only really need 8-bytes there as we only use
> the very first entry of the page table. But we allocate whole page
> anyway. We cannot have the code in the same because, there's hazard that
> a CPU would read page table speculatively and get confused seeing
> garbage.
> 
> This patch introduces place_trampoline() that finds right spot in lower
> memory for trampoline, copies trampoline code there and setups new top
> level page table for 5-level paging.
> 
> At this point we do all the preparation, but not yet use trampoline.
> It will be done in following patch.
> 
> Signed-off-by: Kirill A. Shutemov 
> ---
>  arch/x86/boot/compressed/head_64.S   | 13 +++
>  arch/x86/boot/compressed/pagetable.c | 42 
> 
>  arch/x86/boot/compressed/pagetable.h | 18 
>  3 files changed, 73 insertions(+)
>  create mode 100644 arch/x86/boot/compressed/pagetable.h
> 
> diff --git a/arch/x86/boot/compressed/head_64.S 
> b/arch/x86/boot/compressed/head_64.S
> index 6ac8239af2b6..4d1555b39de0 100644
> --- a/arch/x86/boot/compressed/head_64.S
> +++ b/arch/x86/boot/compressed/head_64.S
> @@ -315,6 +315,18 @@ ENTRY(startup_64)
>* The first step is go into compatibility mode.
>*/
>  
> + /*
> +  * Find suitable place for trampoline and populate it.
> +  * The address will be stored in RCX.
> +  *
> +  * RSI holds real mode data and need to be preserved across
> +  * a function call.
> +  */
> + pushq   %rsi
> + callplace_trampoline
> + popq%rsi
> + movq%rax, %rcx
> +
>   /* Clear additional page table */
>   leaqlvl5_pgtable(%rbx), %rdi
>   xorq%rax, %rax

One request: it's always going to be fragile if the _only_ thing that uses the 
trampoline is the 5-level paging code.

Could we use the trampoline in the 4-level paging case too? It's not required, 
but 
would test much of the trampoline allocation and copying machinery - and the 
performance cost is negligible.

Thanks,

Ingo

Re: [PATCH 3/4] x86/boot/compressed/64: Introduce place_trampoline()

[Ingo Molnar]

* Kirill A. Shutemov  wrote:

> If bootloader enables 64-bit mode with 4-level paging, we need to
> switch over to 5-level paging. The switching requires disabling paging.
> It works fine if kernel itself is loaded below 4G.
> 
> If bootloader put the kernel above 4G (not sure if anybody does this),
> we would loose control as soon as paging is disabled as code becomes
> unreachable.
> 
> To handle the situation, we need a trampoline in lower memory that would
> take care about switching on 5-level paging.
> 
> Apart from trampoline itself we also need place to store top level page
> table in lower memory as we don't have a way to load 64-bit value into
> CR3 from 32-bit mode. We only really need 8-bytes there as we only use
> the very first entry of the page table. But we allocate whole page
> anyway. We cannot have the code in the same because, there's hazard that
> a CPU would read page table speculatively and get confused seeing
> garbage.
> 
> This patch introduces place_trampoline() that finds right spot in lower
> memory for trampoline, copies trampoline code there and setups new top
> level page table for 5-level paging.
> 
> At this point we do all the preparation, but not yet use trampoline.
> It will be done in following patch.
> 
> Signed-off-by: Kirill A. Shutemov 
> ---
>  arch/x86/boot/compressed/head_64.S   | 13 +++
>  arch/x86/boot/compressed/pagetable.c | 42 
> 
>  arch/x86/boot/compressed/pagetable.h | 18 
>  3 files changed, 73 insertions(+)
>  create mode 100644 arch/x86/boot/compressed/pagetable.h

Ok, I like it how almost all of the complexity is now at the .c level.

I'm wondering, how did you test it if no current bootloader puts the kernel to 
above addresses of 4G?

Thanks,

Ingo

Re: [PATCH 3/4] x86/boot/compressed/64: Introduce place_trampoline()

[Ingo Molnar]

* Kirill A. Shutemov  wrote:

> If bootloader enables 64-bit mode with 4-level paging, we need to
> switch over to 5-level paging. The switching requires disabling paging.
> It works fine if kernel itself is loaded below 4G.
> 
> If bootloader put the kernel above 4G (not sure if anybody does this),
> we would loose control as soon as paging is disabled as code becomes
> unreachable.
> 
> To handle the situation, we need a trampoline in lower memory that would
> take care about switching on 5-level paging.
> 
> Apart from trampoline itself we also need place to store top level page
> table in lower memory as we don't have a way to load 64-bit value into
> CR3 from 32-bit mode. We only really need 8-bytes there as we only use
> the very first entry of the page table. But we allocate whole page
> anyway. We cannot have the code in the same because, there's hazard that
> a CPU would read page table speculatively and get confused seeing
> garbage.
> 
> This patch introduces place_trampoline() that finds right spot in lower
> memory for trampoline, copies trampoline code there and setups new top
> level page table for 5-level paging.
> 
> At this point we do all the preparation, but not yet use trampoline.
> It will be done in following patch.
> 
> Signed-off-by: Kirill A. Shutemov 
> ---
>  arch/x86/boot/compressed/head_64.S   | 13 +++
>  arch/x86/boot/compressed/pagetable.c | 42 
> 
>  arch/x86/boot/compressed/pagetable.h | 18 
>  3 files changed, 73 insertions(+)
>  create mode 100644 arch/x86/boot/compressed/pagetable.h

Ok, I like it how almost all of the complexity is now at the .c level.

I'm wondering, how did you test it if no current bootloader puts the kernel to 
above addresses of 4G?

Thanks,

Ingo

[PATCH 3/4] x86/boot/compressed/64: Introduce place_trampoline()

[Kirill A. Shutemov]

If bootloader enables 64-bit mode with 4-level paging, we need to
switch over to 5-level paging. The switching requires disabling paging.
It works fine if kernel itself is loaded below 4G.

If bootloader put the kernel above 4G (not sure if anybody does this),
we would loose control as soon as paging is disabled as code becomes
unreachable.

To handle the situation, we need a trampoline in lower memory that would
take care about switching on 5-level paging.

Apart from trampoline itself we also need place to store top level page
table in lower memory as we don't have a way to load 64-bit value into
CR3 from 32-bit mode. We only really need 8-bytes there as we only use
the very first entry of the page table. But we allocate whole page
anyway. We cannot have the code in the same because, there's hazard that
a CPU would read page table speculatively and get confused seeing
garbage.

This patch introduces place_trampoline() that finds right spot in lower
memory for trampoline, copies trampoline code there and setups new top
level page table for 5-level paging.

At this point we do all the preparation, but not yet use trampoline.
It will be done in following patch.

Signed-off-by: Kirill A. Shutemov 
---
 arch/x86/boot/compressed/head_64.S   | 13 +++
 arch/x86/boot/compressed/pagetable.c | 42 
 arch/x86/boot/compressed/pagetable.h | 18 
 3 files changed, 73 insertions(+)
 create mode 100644 arch/x86/boot/compressed/pagetable.h

diff --git a/arch/x86/boot/compressed/head_64.S 
b/arch/x86/boot/compressed/head_64.S
index 6ac8239af2b6..4d1555b39de0 100644
--- a/arch/x86/boot/compressed/head_64.S
+++ b/arch/x86/boot/compressed/head_64.S
@@ -315,6 +315,18 @@ ENTRY(startup_64)
 * The first step is go into compatibility mode.
 */
 
+   /*
+* Find suitable place for trampoline and populate it.
+* The address will be stored in RCX.
+*
+* RSI holds real mode data and need to be preserved across
+* a function call.
+*/
+   pushq   %rsi
+   callplace_trampoline
+   popq%rsi
+   movq%rax, %rcx
+
/* Clear additional page table */
leaqlvl5_pgtable(%rbx), %rdi
xorq%rax, %rax
@@ -474,6 +486,7 @@ relocated:
 
.code32
 #ifdef CONFIG_X86_5LEVEL
+ENTRY(lvl5_trampoline_src)
 compatible_mode:
/* Setup data and stack segments */
movl$__KERNEL_DS, %eax
diff --git a/arch/x86/boot/compressed/pagetable.c 
b/arch/x86/boot/compressed/pagetable.c
index cd2dd49333cc..74245e9e875f 100644
--- a/arch/x86/boot/compressed/pagetable.c
+++ b/arch/x86/boot/compressed/pagetable.c
@@ -23,6 +23,8 @@
 #undef CONFIG_AMD_MEM_ENCRYPT
 
 #include "misc.h"
+#include "pagetable.h"
+#include "../string.h"
 
 /* These actually do the work of building the kernel identity maps. */
 #include 
@@ -172,4 +174,44 @@ int need_to_enabled_l5(void)
 
return 1;
 }
+
+#define BIOS_START_MIN 0x2U/* 128K, less than this is 
insane */
+#define BIOS_START_MAX 0x9f000U/* 640K, absolute maximum */
+
+unsigned long *place_trampoline(void)
+{
+   unsigned long bios_start, ebda_start, trampoline_start, *trampoline;
+
+   /* Based on reserve_bios_regions() */
+
+   ebda_start = *(unsigned short *)0x40e << 4;
+   bios_start = *(unsigned short *)0x413 << 10;
+
+   if (bios_start < BIOS_START_MIN || bios_start > BIOS_START_MAX)
+   bios_start = BIOS_START_MAX;
+
+   if (ebda_start > BIOS_START_MIN && ebda_start < bios_start)
+   bios_start = ebda_start;
+
+   /* Place trampoline below end of low memory, aligned to 4k */
+   trampoline_start = bios_start - LVL5_TRAMPOLINE_SIZE;
+   trampoline_start = round_down(trampoline_start, PAGE_SIZE);
+
+   trampoline = (unsigned long *)trampoline_start;
+
+   /* Clear trampoline memory first */
+   memset(trampoline, 0, LVL5_TRAMPOLINE_SIZE);
+
+   /* Copy trampoline code in place */
+   memcpy(trampoline + LVL5_TRAMPOLINE_CODE_OFF / sizeof(unsigned long),
+   _trampoline_src, LVL5_TRAMPOLINE_CODE_SIZE);
+
+   /*
+* Setup current CR3 as the first and the only entry in a new top level
+* page table.
+*/
+   trampoline[0] = __read_cr3() + _PAGE_TABLE_NOENC;
+
+   return trampoline;
+}
 #endif
diff --git a/arch/x86/boot/compressed/pagetable.h 
b/arch/x86/boot/compressed/pagetable.h
new file mode 100644
index ..906436cc1c02
--- /dev/null
+++ b/arch/x86/boot/compressed/pagetable.h
@@ -0,0 +1,18 @@
+#ifndef BOOT_COMPRESSED_PAGETABLE_H
+#define BOOT_COMPRESSED_PAGETABLE_H
+
+#define LVL5_TRAMPOLINE_SIZE   (2 * PAGE_SIZE)
+
+#define LVL5_TRAMPOLINE_PGTABLE_OFF0
+
+#define LVL5_TRAMPOLINE_CODE_OFF   PAGE_SIZE
+#define LVL5_TRAMPOLINE_CODE_SIZE  0x40
+
+#define LVL5_TRAMPOLINE_STACK_END  

[PATCH 3/4] x86/boot/compressed/64: Introduce place_trampoline()

[Kirill A. Shutemov]

If bootloader enables 64-bit mode with 4-level paging, we need to
switch over to 5-level paging. The switching requires disabling paging.
It works fine if kernel itself is loaded below 4G.

If bootloader put the kernel above 4G (not sure if anybody does this),
we would loose control as soon as paging is disabled as code becomes
unreachable.

To handle the situation, we need a trampoline in lower memory that would
take care about switching on 5-level paging.

Apart from trampoline itself we also need place to store top level page
table in lower memory as we don't have a way to load 64-bit value into
CR3 from 32-bit mode. We only really need 8-bytes there as we only use
the very first entry of the page table. But we allocate whole page
anyway. We cannot have the code in the same because, there's hazard that
a CPU would read page table speculatively and get confused seeing
garbage.

This patch introduces place_trampoline() that finds right spot in lower
memory for trampoline, copies trampoline code there and setups new top
level page table for 5-level paging.

At this point we do all the preparation, but not yet use trampoline.
It will be done in following patch.

Signed-off-by: Kirill A. Shutemov 
---
 arch/x86/boot/compressed/head_64.S   | 13 +++
 arch/x86/boot/compressed/pagetable.c | 42 
 arch/x86/boot/compressed/pagetable.h | 18 
 3 files changed, 73 insertions(+)
 create mode 100644 arch/x86/boot/compressed/pagetable.h

diff --git a/arch/x86/boot/compressed/head_64.S 
b/arch/x86/boot/compressed/head_64.S
index 6ac8239af2b6..4d1555b39de0 100644
--- a/arch/x86/boot/compressed/head_64.S
+++ b/arch/x86/boot/compressed/head_64.S
@@ -315,6 +315,18 @@ ENTRY(startup_64)
 * The first step is go into compatibility mode.
 */
 
+   /*
+* Find suitable place for trampoline and populate it.
+* The address will be stored in RCX.
+*
+* RSI holds real mode data and need to be preserved across
+* a function call.
+*/
+   pushq   %rsi
+   callplace_trampoline
+   popq%rsi
+   movq%rax, %rcx
+
/* Clear additional page table */
leaqlvl5_pgtable(%rbx), %rdi
xorq%rax, %rax
@@ -474,6 +486,7 @@ relocated:
 
.code32
 #ifdef CONFIG_X86_5LEVEL
+ENTRY(lvl5_trampoline_src)
 compatible_mode:
/* Setup data and stack segments */
movl$__KERNEL_DS, %eax
diff --git a/arch/x86/boot/compressed/pagetable.c 
b/arch/x86/boot/compressed/pagetable.c
index cd2dd49333cc..74245e9e875f 100644
--- a/arch/x86/boot/compressed/pagetable.c
+++ b/arch/x86/boot/compressed/pagetable.c
@@ -23,6 +23,8 @@
 #undef CONFIG_AMD_MEM_ENCRYPT
 
 #include "misc.h"
+#include "pagetable.h"
+#include "../string.h"
 
 /* These actually do the work of building the kernel identity maps. */
 #include 
@@ -172,4 +174,44 @@ int need_to_enabled_l5(void)
 
return 1;
 }
+
+#define BIOS_START_MIN 0x2U/* 128K, less than this is 
insane */
+#define BIOS_START_MAX 0x9f000U/* 640K, absolute maximum */
+
+unsigned long *place_trampoline(void)
+{
+   unsigned long bios_start, ebda_start, trampoline_start, *trampoline;
+
+   /* Based on reserve_bios_regions() */
+
+   ebda_start = *(unsigned short *)0x40e << 4;
+   bios_start = *(unsigned short *)0x413 << 10;
+
+   if (bios_start < BIOS_START_MIN || bios_start > BIOS_START_MAX)
+   bios_start = BIOS_START_MAX;
+
+   if (ebda_start > BIOS_START_MIN && ebda_start < bios_start)
+   bios_start = ebda_start;
+
+   /* Place trampoline below end of low memory, aligned to 4k */
+   trampoline_start = bios_start - LVL5_TRAMPOLINE_SIZE;
+   trampoline_start = round_down(trampoline_start, PAGE_SIZE);
+
+   trampoline = (unsigned long *)trampoline_start;
+
+   /* Clear trampoline memory first */
+   memset(trampoline, 0, LVL5_TRAMPOLINE_SIZE);
+
+   /* Copy trampoline code in place */
+   memcpy(trampoline + LVL5_TRAMPOLINE_CODE_OFF / sizeof(unsigned long),
+   _trampoline_src, LVL5_TRAMPOLINE_CODE_SIZE);
+
+   /*
+* Setup current CR3 as the first and the only entry in a new top level
+* page table.
+*/
+   trampoline[0] = __read_cr3() + _PAGE_TABLE_NOENC;
+
+   return trampoline;
+}
 #endif
diff --git a/arch/x86/boot/compressed/pagetable.h 
b/arch/x86/boot/compressed/pagetable.h
new file mode 100644
index ..906436cc1c02
--- /dev/null
+++ b/arch/x86/boot/compressed/pagetable.h
@@ -0,0 +1,18 @@
+#ifndef BOOT_COMPRESSED_PAGETABLE_H
+#define BOOT_COMPRESSED_PAGETABLE_H
+
+#define LVL5_TRAMPOLINE_SIZE   (2 * PAGE_SIZE)
+
+#define LVL5_TRAMPOLINE_PGTABLE_OFF0
+
+#define LVL5_TRAMPOLINE_CODE_OFF   PAGE_SIZE
+#define LVL5_TRAMPOLINE_CODE_SIZE  0x40
+
+#define LVL5_TRAMPOLINE_STACK_END  LVL5_TRAMPOLINE_SIZE
+
+#ifndef __ASSEMBLER__

[PATCH 3/4] x86/boot/compressed/64: Introduce place_trampoline()

[Kirill A. Shutemov]

If bootloader enables 64-bit mode with 4-level paging, we need to
switch over to 5-level paging. The switching requires disabling paging.
It works fine if kernel itself is loaded below 4G.

If bootloader put the kernel above 4G (not sure if anybody does this),
we would loose control as soon as paging is disabled as code becomes
unreachable.

To handle the situation, we need a trampoline in lower memory that would
take care about switching on 5-level paging.

Apart from trampoline itself we also need place to store top level page
table in lower memory as we don't have a way to load 64-bit value into
CR3 from 32-bit mode. We only really need 8-bytes there as we only use
the very first entry of the page table. But we allocate whole page
anyway. We cannot have the code in the same because, there's hazard that
a CPU would read page table speculatively and get confused seeing
garbage.

This patch introduces place_trampoline() that finds right spot in lower
memory for trampoline, copies trampoline code there and setups new top
level page table for 5-level paging.

At this point we do all the preparation, but not yet use trampoline.
It will be done in following patch.

Signed-off-by: Kirill A. Shutemov 
---
 arch/x86/boot/compressed/head_64.S   | 13 +++
 arch/x86/boot/compressed/pagetable.c | 42 
 arch/x86/boot/compressed/pagetable.h | 18 
 3 files changed, 73 insertions(+)
 create mode 100644 arch/x86/boot/compressed/pagetable.h

diff --git a/arch/x86/boot/compressed/head_64.S 
b/arch/x86/boot/compressed/head_64.S
index 6ac8239af2b6..4d1555b39de0 100644
--- a/arch/x86/boot/compressed/head_64.S
+++ b/arch/x86/boot/compressed/head_64.S
@@ -315,6 +315,18 @@ ENTRY(startup_64)
 * The first step is go into compatibility mode.
 */
 
+   /*
+* Find suitable place for trampoline and populate it.
+* The address will be stored in RCX.
+*
+* RSI holds real mode data and need to be preserved across
+* a function call.
+*/
+   pushq   %rsi
+   callplace_trampoline
+   popq%rsi
+   movq%rax, %rcx
+
/* Clear additional page table */
leaqlvl5_pgtable(%rbx), %rdi
xorq%rax, %rax
@@ -474,6 +486,7 @@ relocated:
 
.code32
 #ifdef CONFIG_X86_5LEVEL
+ENTRY(lvl5_trampoline_src)
 compatible_mode:
/* Setup data and stack segments */
movl$__KERNEL_DS, %eax
diff --git a/arch/x86/boot/compressed/pagetable.c 
b/arch/x86/boot/compressed/pagetable.c
index 76d25a82e3ac..a04fb69a453f 100644
--- a/arch/x86/boot/compressed/pagetable.c
+++ b/arch/x86/boot/compressed/pagetable.c
@@ -23,6 +23,8 @@
 #undef CONFIG_AMD_MEM_ENCRYPT
 
 #include "misc.h"
+#include "pagetable.h"
+#include "../string.h"
 
 /* These actually do the work of building the kernel identity maps. */
 #include 
@@ -167,4 +169,44 @@ int need_to_enabled_l5(void)
 
return 1;
 }
+
+#define BIOS_START_MIN 0x2U/* 128K, less than this is 
insane */
+#define BIOS_START_MAX 0x9f000U/* 640K, absolute maximum */
+
+unsigned long *place_trampoline()
+{
+   unsigned long bios_start, ebda_start, trampoline_start, *trampoline;
+
+   /* Based on reserve_bios_regions() */
+
+   ebda_start = *(unsigned short *)0x40e << 4;
+   bios_start = *(unsigned short *)0x413 << 10;
+
+   if (bios_start < BIOS_START_MIN || bios_start > BIOS_START_MAX)
+   bios_start = BIOS_START_MAX;
+
+   if (ebda_start > BIOS_START_MIN && ebda_start < bios_start)
+   bios_start = ebda_start;
+
+   /* Place trampoline below end of low memory, aligned to 4k */
+   trampoline_start = bios_start - LVL5_TRAMPOLINE_SIZE;
+   trampoline_start = round_down(trampoline_start, PAGE_SIZE);
+
+   trampoline = (unsigned long *)trampoline_start;
+
+   /* Clear trampoline memory first */
+   memset(trampoline, 0, LVL5_TRAMPOLINE_SIZE);
+
+   /* Copy trampoline code in place */
+   memcpy(trampoline + LVL5_TRAMPOLINE_CODE_OFF / sizeof(unsigned long),
+   _trampoline_src, LVL5_TRAMPOLINE_CODE_SIZE);
+
+   /*
+* Setup current CR3 as the first and the only entry in a new top level
+* page table.
+*/
+   trampoline[0] = __read_cr3() + _PAGE_TABLE_NOENC;
+
+   return trampoline;
+}
 #endif
diff --git a/arch/x86/boot/compressed/pagetable.h 
b/arch/x86/boot/compressed/pagetable.h
new file mode 100644
index ..906436cc1c02
--- /dev/null
+++ b/arch/x86/boot/compressed/pagetable.h
@@ -0,0 +1,18 @@
+#ifndef BOOT_COMPRESSED_PAGETABLE_H
+#define BOOT_COMPRESSED_PAGETABLE_H
+
+#define LVL5_TRAMPOLINE_SIZE   (2 * PAGE_SIZE)
+
+#define LVL5_TRAMPOLINE_PGTABLE_OFF0
+
+#define LVL5_TRAMPOLINE_CODE_OFF   PAGE_SIZE
+#define LVL5_TRAMPOLINE_CODE_SIZE  0x40
+
+#define LVL5_TRAMPOLINE_STACK_END  

[PATCH 3/4] x86/boot/compressed/64: Introduce place_trampoline()

[Kirill A. Shutemov]

If bootloader enables 64-bit mode with 4-level paging, we need to
switch over to 5-level paging. The switching requires disabling paging.
It works fine if kernel itself is loaded below 4G.

If bootloader put the kernel above 4G (not sure if anybody does this),
we would loose control as soon as paging is disabled as code becomes
unreachable.

To handle the situation, we need a trampoline in lower memory that would
take care about switching on 5-level paging.

Apart from trampoline itself we also need place to store top level page
table in lower memory as we don't have a way to load 64-bit value into
CR3 from 32-bit mode. We only really need 8-bytes there as we only use
the very first entry of the page table. But we allocate whole page
anyway. We cannot have the code in the same because, there's hazard that
a CPU would read page table speculatively and get confused seeing
garbage.

This patch introduces place_trampoline() that finds right spot in lower
memory for trampoline, copies trampoline code there and setups new top
level page table for 5-level paging.

At this point we do all the preparation, but not yet use trampoline.
It will be done in following patch.

Signed-off-by: Kirill A. Shutemov 
---
 arch/x86/boot/compressed/head_64.S   | 13 +++
 arch/x86/boot/compressed/pagetable.c | 42 
 arch/x86/boot/compressed/pagetable.h | 18 
 3 files changed, 73 insertions(+)
 create mode 100644 arch/x86/boot/compressed/pagetable.h

diff --git a/arch/x86/boot/compressed/head_64.S 
b/arch/x86/boot/compressed/head_64.S
index 6ac8239af2b6..4d1555b39de0 100644
--- a/arch/x86/boot/compressed/head_64.S
+++ b/arch/x86/boot/compressed/head_64.S
@@ -315,6 +315,18 @@ ENTRY(startup_64)
 * The first step is go into compatibility mode.
 */
 
+   /*
+* Find suitable place for trampoline and populate it.
+* The address will be stored in RCX.
+*
+* RSI holds real mode data and need to be preserved across
+* a function call.
+*/
+   pushq   %rsi
+   callplace_trampoline
+   popq%rsi
+   movq%rax, %rcx
+
/* Clear additional page table */
leaqlvl5_pgtable(%rbx), %rdi
xorq%rax, %rax
@@ -474,6 +486,7 @@ relocated:
 
.code32
 #ifdef CONFIG_X86_5LEVEL
+ENTRY(lvl5_trampoline_src)
 compatible_mode:
/* Setup data and stack segments */
movl$__KERNEL_DS, %eax
diff --git a/arch/x86/boot/compressed/pagetable.c 
b/arch/x86/boot/compressed/pagetable.c
index 76d25a82e3ac..a04fb69a453f 100644
--- a/arch/x86/boot/compressed/pagetable.c
+++ b/arch/x86/boot/compressed/pagetable.c
@@ -23,6 +23,8 @@
 #undef CONFIG_AMD_MEM_ENCRYPT
 
 #include "misc.h"
+#include "pagetable.h"
+#include "../string.h"
 
 /* These actually do the work of building the kernel identity maps. */
 #include 
@@ -167,4 +169,44 @@ int need_to_enabled_l5(void)
 
return 1;
 }
+
+#define BIOS_START_MIN 0x2U/* 128K, less than this is 
insane */
+#define BIOS_START_MAX 0x9f000U/* 640K, absolute maximum */
+
+unsigned long *place_trampoline()
+{
+   unsigned long bios_start, ebda_start, trampoline_start, *trampoline;
+
+   /* Based on reserve_bios_regions() */
+
+   ebda_start = *(unsigned short *)0x40e << 4;
+   bios_start = *(unsigned short *)0x413 << 10;
+
+   if (bios_start < BIOS_START_MIN || bios_start > BIOS_START_MAX)
+   bios_start = BIOS_START_MAX;
+
+   if (ebda_start > BIOS_START_MIN && ebda_start < bios_start)
+   bios_start = ebda_start;
+
+   /* Place trampoline below end of low memory, aligned to 4k */
+   trampoline_start = bios_start - LVL5_TRAMPOLINE_SIZE;
+   trampoline_start = round_down(trampoline_start, PAGE_SIZE);
+
+   trampoline = (unsigned long *)trampoline_start;
+
+   /* Clear trampoline memory first */
+   memset(trampoline, 0, LVL5_TRAMPOLINE_SIZE);
+
+   /* Copy trampoline code in place */
+   memcpy(trampoline + LVL5_TRAMPOLINE_CODE_OFF / sizeof(unsigned long),
+   _trampoline_src, LVL5_TRAMPOLINE_CODE_SIZE);
+
+   /*
+* Setup current CR3 as the first and the only entry in a new top level
+* page table.
+*/
+   trampoline[0] = __read_cr3() + _PAGE_TABLE_NOENC;
+
+   return trampoline;
+}
 #endif
diff --git a/arch/x86/boot/compressed/pagetable.h 
b/arch/x86/boot/compressed/pagetable.h
new file mode 100644
index ..906436cc1c02
--- /dev/null
+++ b/arch/x86/boot/compressed/pagetable.h
@@ -0,0 +1,18 @@
+#ifndef BOOT_COMPRESSED_PAGETABLE_H
+#define BOOT_COMPRESSED_PAGETABLE_H
+
+#define LVL5_TRAMPOLINE_SIZE   (2 * PAGE_SIZE)
+
+#define LVL5_TRAMPOLINE_PGTABLE_OFF0
+
+#define LVL5_TRAMPOLINE_CODE_OFF   PAGE_SIZE
+#define LVL5_TRAMPOLINE_CODE_SIZE  0x40
+
+#define LVL5_TRAMPOLINE_STACK_END  LVL5_TRAMPOLINE_SIZE
+
+#ifndef __ASSEMBLER__
+