Re: KASAN vs vmapped stacks

[Dmitry Vyukov]

On Mon, Jul 11, 2016 at 4:10 PM, Andrey Ryabinin
 wrote:
>
>
> On 07/11/2016 01:29 PM, Dmitry Vyukov wrote:
>> On Mon, Jul 11, 2016 at 11:57 AM, Andrey Ryabinin
>>  wrote:
>>>
>>>
>>> On 07/10/2016 03:47 PM, Andy Lutomirski wrote:
 Hi all-

 I found two nasty issues with virtually mapped stacks if KASAN is
 enabled.  The first issue is a crash: the first non-init stack is
 allocated and accessed before KASAN initializes its zero shadow
 AFAICT, which means that we switch to that stack and then blow up when
 we start recursively faulting on failed accesses to the shadow.

>>>
>>> KASAN initialized quite early, before any non-init task exists. The crash 
>>> happens
>>> because non-init task writes to write-protected zero shadow.
>>> Currently KASAN doesn't allocate shadow memory for vmalloc addresses, we 
>>> just map single
>>> zero page and write protect it.
>>>
>>>
 The second issue is that, even if we survive (we initialize the zero
 shadow on time), KASAN will fail to protect hte stack.

 For now, I just disabled use of virtually mapped stacks if KASAN is
 on.  Do you have any easy ideas to fix it?

>>>
>>> Allocate shadow memory which backs vmalloc/vmap allocations is the only way 
>>> to fix this.
>>> I can do this, and post the patches soon enough.
>>
>> Do you want to allocate it eagerly? Won't it consume 1/8 of vmalloc
>> range worth of physical memory?
>>
>
> I have no idea what do you mean by eager allocation.


For physical memory we preallocate whole shadow range at startup. This
consumes 1/8 of physical memory. That's what I meant by eager
allocation. Lazy allocation would be to allocate shadow for memory
when it's first allocated/used.

> What we need is to allocate shadow per vmalloc()/vmap() call if such 
> allocation requires non-zero shadow, like stacks or modules.
> Sometimes we certainly know that the whole shadow page will contain only 
> zeros. In such cases we can use zero page to not waste memory.

Do you want to add a gfp_t flag that says "this range requires writable shadow"?
Note that shadow for vmalloc memory will consume from 2/8 to 9/8,
because we back both virtual and physical addresses and because shadow
can be allocated only with page granularity. That's probably fine if
it is used only for kernel stacks.

Re: KASAN vs vmapped stacks

[Dmitry Vyukov]

On Mon, Jul 11, 2016 at 4:10 PM, Andrey Ryabinin
 wrote:
>
>
> On 07/11/2016 01:29 PM, Dmitry Vyukov wrote:
>> On Mon, Jul 11, 2016 at 11:57 AM, Andrey Ryabinin
>>  wrote:
>>>
>>>
>>> On 07/10/2016 03:47 PM, Andy Lutomirski wrote:
 Hi all-

 I found two nasty issues with virtually mapped stacks if KASAN is
 enabled.  The first issue is a crash: the first non-init stack is
 allocated and accessed before KASAN initializes its zero shadow
 AFAICT, which means that we switch to that stack and then blow up when
 we start recursively faulting on failed accesses to the shadow.

>>>
>>> KASAN initialized quite early, before any non-init task exists. The crash 
>>> happens
>>> because non-init task writes to write-protected zero shadow.
>>> Currently KASAN doesn't allocate shadow memory for vmalloc addresses, we 
>>> just map single
>>> zero page and write protect it.
>>>
>>>
 The second issue is that, even if we survive (we initialize the zero
 shadow on time), KASAN will fail to protect hte stack.

 For now, I just disabled use of virtually mapped stacks if KASAN is
 on.  Do you have any easy ideas to fix it?

>>>
>>> Allocate shadow memory which backs vmalloc/vmap allocations is the only way 
>>> to fix this.
>>> I can do this, and post the patches soon enough.
>>
>> Do you want to allocate it eagerly? Won't it consume 1/8 of vmalloc
>> range worth of physical memory?
>>
>
> I have no idea what do you mean by eager allocation.


For physical memory we preallocate whole shadow range at startup. This
consumes 1/8 of physical memory. That's what I meant by eager
allocation. Lazy allocation would be to allocate shadow for memory
when it's first allocated/used.

> What we need is to allocate shadow per vmalloc()/vmap() call if such 
> allocation requires non-zero shadow, like stacks or modules.
> Sometimes we certainly know that the whole shadow page will contain only 
> zeros. In such cases we can use zero page to not waste memory.

Do you want to add a gfp_t flag that says "this range requires writable shadow"?
Note that shadow for vmalloc memory will consume from 2/8 to 9/8,
because we back both virtual and physical addresses and because shadow
can be allocated only with page granularity. That's probably fine if
it is used only for kernel stacks.

Re: KASAN vs vmapped stacks

[Andrey Ryabinin]

On 07/11/2016 01:29 PM, Dmitry Vyukov wrote:
> On Mon, Jul 11, 2016 at 11:57 AM, Andrey Ryabinin
>  wrote:
>>
>>
>> On 07/10/2016 03:47 PM, Andy Lutomirski wrote:
>>> Hi all-
>>>
>>> I found two nasty issues with virtually mapped stacks if KASAN is
>>> enabled.  The first issue is a crash: the first non-init stack is
>>> allocated and accessed before KASAN initializes its zero shadow
>>> AFAICT, which means that we switch to that stack and then blow up when
>>> we start recursively faulting on failed accesses to the shadow.
>>>
>>
>> KASAN initialized quite early, before any non-init task exists. The crash 
>> happens
>> because non-init task writes to write-protected zero shadow.
>> Currently KASAN doesn't allocate shadow memory for vmalloc addresses, we 
>> just map single
>> zero page and write protect it.
>>
>>
>>> The second issue is that, even if we survive (we initialize the zero
>>> shadow on time), KASAN will fail to protect hte stack.
>>>
>>> For now, I just disabled use of virtually mapped stacks if KASAN is
>>> on.  Do you have any easy ideas to fix it?
>>>
>>
>> Allocate shadow memory which backs vmalloc/vmap allocations is the only way 
>> to fix this.
>> I can do this, and post the patches soon enough.
> 
> Do you want to allocate it eagerly? Won't it consume 1/8 of vmalloc
> range worth of physical memory?
> 

I have no idea what do you mean by eager allocation.
What we need is to allocate shadow per vmalloc()/vmap() call if such allocation 
requires non-zero shadow, like stacks or modules.
Sometimes we certainly know that the whole shadow page will contain only zeros. 
In such cases we can use zero page to not waste memory.

Re: KASAN vs vmapped stacks

[Andrey Ryabinin]

On 07/11/2016 01:29 PM, Dmitry Vyukov wrote:
> On Mon, Jul 11, 2016 at 11:57 AM, Andrey Ryabinin
>  wrote:
>>
>>
>> On 07/10/2016 03:47 PM, Andy Lutomirski wrote:
>>> Hi all-
>>>
>>> I found two nasty issues with virtually mapped stacks if KASAN is
>>> enabled.  The first issue is a crash: the first non-init stack is
>>> allocated and accessed before KASAN initializes its zero shadow
>>> AFAICT, which means that we switch to that stack and then blow up when
>>> we start recursively faulting on failed accesses to the shadow.
>>>
>>
>> KASAN initialized quite early, before any non-init task exists. The crash 
>> happens
>> because non-init task writes to write-protected zero shadow.
>> Currently KASAN doesn't allocate shadow memory for vmalloc addresses, we 
>> just map single
>> zero page and write protect it.
>>
>>
>>> The second issue is that, even if we survive (we initialize the zero
>>> shadow on time), KASAN will fail to protect hte stack.
>>>
>>> For now, I just disabled use of virtually mapped stacks if KASAN is
>>> on.  Do you have any easy ideas to fix it?
>>>
>>
>> Allocate shadow memory which backs vmalloc/vmap allocations is the only way 
>> to fix this.
>> I can do this, and post the patches soon enough.
> 
> Do you want to allocate it eagerly? Won't it consume 1/8 of vmalloc
> range worth of physical memory?
> 

I have no idea what do you mean by eager allocation.
What we need is to allocate shadow per vmalloc()/vmap() call if such allocation 
requires non-zero shadow, like stacks or modules.
Sometimes we certainly know that the whole shadow page will contain only zeros. 
In such cases we can use zero page to not waste memory.

Re: KASAN vs vmapped stacks

[Andy Lutomirski]

On Mon, Jul 11, 2016 at 1:30 AM, Dmitry Vyukov  wrote:
> On Sun, Jul 10, 2016 at 2:47 PM, Andy Lutomirski  wrote:
>> Hi all-
>>
>> I found two nasty issues with virtually mapped stacks if KASAN is
>> enabled.  The first issue is a crash: the first non-init stack is
>> allocated and accessed before KASAN initializes its zero shadow
>> AFAICT, which means that we switch to that stack and then blow up when
>> we start recursively faulting on failed accesses to the shadow.
>>
>> The second issue is that, even if we survive (we initialize the zero
>> shadow on time), KASAN will fail to protect hte stack.
>>
>> For now, I just disabled use of virtually mapped stacks if KASAN is
>> on.  Do you have any easy ideas to fix it?
>
>
> Hi Andy,
>
> What is "virtually mapped stacks"? How are they enabled?

It's a patchset I'm working on, and I'm hoping to get them in to 4.8.
They're only supported (so far) on x86, and they're enabled by a
config option.  The init task has a conventional stack, but all other
stacks are vmalloced.

> KASAN does not have real shadow for vmalloc range. Fixing this will
> probably involve allocating real shadow memory for that range which
> will increase memory consumption.
> You said that you disabled the virtually mapped stacks, which means
> that it is not a critical feature for your environment. When is it
> supposed to be used? What are the benefits? For now I am leaning
> towards automatically disabling virtually mapped stacks when KASAN is
> enabled.

That's what I'm doing right now.  Enabling virtually mapped stacks
gives reliably stack overflow detection and avoids needing
higher-order pages.  It's certainly not a critical feature, but it's
nice, and supporting both it and KASAN at the same time would be nice.

--Andy

Re: KASAN vs vmapped stacks

[Andy Lutomirski]

On Mon, Jul 11, 2016 at 1:30 AM, Dmitry Vyukov  wrote:
> On Sun, Jul 10, 2016 at 2:47 PM, Andy Lutomirski  wrote:
>> Hi all-
>>
>> I found two nasty issues with virtually mapped stacks if KASAN is
>> enabled.  The first issue is a crash: the first non-init stack is
>> allocated and accessed before KASAN initializes its zero shadow
>> AFAICT, which means that we switch to that stack and then blow up when
>> we start recursively faulting on failed accesses to the shadow.
>>
>> The second issue is that, even if we survive (we initialize the zero
>> shadow on time), KASAN will fail to protect hte stack.
>>
>> For now, I just disabled use of virtually mapped stacks if KASAN is
>> on.  Do you have any easy ideas to fix it?
>
>
> Hi Andy,
>
> What is "virtually mapped stacks"? How are they enabled?

It's a patchset I'm working on, and I'm hoping to get them in to 4.8.
They're only supported (so far) on x86, and they're enabled by a
config option.  The init task has a conventional stack, but all other
stacks are vmalloced.

> KASAN does not have real shadow for vmalloc range. Fixing this will
> probably involve allocating real shadow memory for that range which
> will increase memory consumption.
> You said that you disabled the virtually mapped stacks, which means
> that it is not a critical feature for your environment. When is it
> supposed to be used? What are the benefits? For now I am leaning
> towards automatically disabling virtually mapped stacks when KASAN is
> enabled.

That's what I'm doing right now.  Enabling virtually mapped stacks
gives reliably stack overflow detection and avoids needing
higher-order pages.  It's certainly not a critical feature, but it's
nice, and supporting both it and KASAN at the same time would be nice.

--Andy

Re: KASAN vs vmapped stacks

[Dmitry Vyukov]

On Mon, Jul 11, 2016 at 11:57 AM, Andrey Ryabinin
 wrote:
>
>
> On 07/10/2016 03:47 PM, Andy Lutomirski wrote:
>> Hi all-
>>
>> I found two nasty issues with virtually mapped stacks if KASAN is
>> enabled.  The first issue is a crash: the first non-init stack is
>> allocated and accessed before KASAN initializes its zero shadow
>> AFAICT, which means that we switch to that stack and then blow up when
>> we start recursively faulting on failed accesses to the shadow.
>>
>
> KASAN initialized quite early, before any non-init task exists. The crash 
> happens
> because non-init task writes to write-protected zero shadow.
> Currently KASAN doesn't allocate shadow memory for vmalloc addresses, we just 
> map single
> zero page and write protect it.
>
>
>> The second issue is that, even if we survive (we initialize the zero
>> shadow on time), KASAN will fail to protect hte stack.
>>
>> For now, I just disabled use of virtually mapped stacks if KASAN is
>> on.  Do you have any easy ideas to fix it?
>>
>
> Allocate shadow memory which backs vmalloc/vmap allocations is the only way 
> to fix this.
> I can do this, and post the patches soon enough.

Do you want to allocate it eagerly? Won't it consume 1/8 of vmalloc
range worth of physical memory?

Re: KASAN vs vmapped stacks

[Dmitry Vyukov]

On Mon, Jul 11, 2016 at 11:57 AM, Andrey Ryabinin
 wrote:
>
>
> On 07/10/2016 03:47 PM, Andy Lutomirski wrote:
>> Hi all-
>>
>> I found two nasty issues with virtually mapped stacks if KASAN is
>> enabled.  The first issue is a crash: the first non-init stack is
>> allocated and accessed before KASAN initializes its zero shadow
>> AFAICT, which means that we switch to that stack and then blow up when
>> we start recursively faulting on failed accesses to the shadow.
>>
>
> KASAN initialized quite early, before any non-init task exists. The crash 
> happens
> because non-init task writes to write-protected zero shadow.
> Currently KASAN doesn't allocate shadow memory for vmalloc addresses, we just 
> map single
> zero page and write protect it.
>
>
>> The second issue is that, even if we survive (we initialize the zero
>> shadow on time), KASAN will fail to protect hte stack.
>>
>> For now, I just disabled use of virtually mapped stacks if KASAN is
>> on.  Do you have any easy ideas to fix it?
>>
>
> Allocate shadow memory which backs vmalloc/vmap allocations is the only way 
> to fix this.
> I can do this, and post the patches soon enough.

Do you want to allocate it eagerly? Won't it consume 1/8 of vmalloc
range worth of physical memory?

Re: KASAN vs vmapped stacks

[Andrey Ryabinin]

On 07/10/2016 03:47 PM, Andy Lutomirski wrote:
> Hi all-
> 
> I found two nasty issues with virtually mapped stacks if KASAN is
> enabled.  The first issue is a crash: the first non-init stack is
> allocated and accessed before KASAN initializes its zero shadow
> AFAICT, which means that we switch to that stack and then blow up when
> we start recursively faulting on failed accesses to the shadow.
> 

KASAN initialized quite early, before any non-init task exists. The crash 
happens
because non-init task writes to write-protected zero shadow.
Currently KASAN doesn't allocate shadow memory for vmalloc addresses, we just 
map single
zero page and write protect it. 


> The second issue is that, even if we survive (we initialize the zero
> shadow on time), KASAN will fail to protect hte stack.
> 
> For now, I just disabled use of virtually mapped stacks if KASAN is
> on.  Do you have any easy ideas to fix it?
> 

Allocate shadow memory which backs vmalloc/vmap allocations is the only way to 
fix this.
I can do this, and post the patches soon enough.


> Thanks,
> Andy
> 

Re: KASAN vs vmapped stacks

[Andrey Ryabinin]

On 07/10/2016 03:47 PM, Andy Lutomirski wrote:
> Hi all-
> 
> I found two nasty issues with virtually mapped stacks if KASAN is
> enabled.  The first issue is a crash: the first non-init stack is
> allocated and accessed before KASAN initializes its zero shadow
> AFAICT, which means that we switch to that stack and then blow up when
> we start recursively faulting on failed accesses to the shadow.
> 

KASAN initialized quite early, before any non-init task exists. The crash 
happens
because non-init task writes to write-protected zero shadow.
Currently KASAN doesn't allocate shadow memory for vmalloc addresses, we just 
map single
zero page and write protect it. 


> The second issue is that, even if we survive (we initialize the zero
> shadow on time), KASAN will fail to protect hte stack.
> 
> For now, I just disabled use of virtually mapped stacks if KASAN is
> on.  Do you have any easy ideas to fix it?
> 

Allocate shadow memory which backs vmalloc/vmap allocations is the only way to 
fix this.
I can do this, and post the patches soon enough.


> Thanks,
> Andy
> 

Re: KASAN vs vmapped stacks

[Dmitry Vyukov]

On Sun, Jul 10, 2016 at 2:47 PM, Andy Lutomirski  wrote:
> Hi all-
>
> I found two nasty issues with virtually mapped stacks if KASAN is
> enabled.  The first issue is a crash: the first non-init stack is
> allocated and accessed before KASAN initializes its zero shadow
> AFAICT, which means that we switch to that stack and then blow up when
> we start recursively faulting on failed accesses to the shadow.
>
> The second issue is that, even if we survive (we initialize the zero
> shadow on time), KASAN will fail to protect hte stack.
>
> For now, I just disabled use of virtually mapped stacks if KASAN is
> on.  Do you have any easy ideas to fix it?


Hi Andy,

What is "virtually mapped stacks"? How are they enabled?
KASAN does not have real shadow for vmalloc range. Fixing this will
probably involve allocating real shadow memory for that range which
will increase memory consumption.
You said that you disabled the virtually mapped stacks, which means
that it is not a critical feature for your environment. When is it
supposed to be used? What are the benefits? For now I am leaning
towards automatically disabling virtually mapped stacks when KASAN is
enabled.

Thanks for the report

Re: KASAN vs vmapped stacks

[Dmitry Vyukov]

On Sun, Jul 10, 2016 at 2:47 PM, Andy Lutomirski  wrote:
> Hi all-
>
> I found two nasty issues with virtually mapped stacks if KASAN is
> enabled.  The first issue is a crash: the first non-init stack is
> allocated and accessed before KASAN initializes its zero shadow
> AFAICT, which means that we switch to that stack and then blow up when
> we start recursively faulting on failed accesses to the shadow.
>
> The second issue is that, even if we survive (we initialize the zero
> shadow on time), KASAN will fail to protect hte stack.
>
> For now, I just disabled use of virtually mapped stacks if KASAN is
> on.  Do you have any easy ideas to fix it?


Hi Andy,

What is "virtually mapped stacks"? How are they enabled?
KASAN does not have real shadow for vmalloc range. Fixing this will
probably involve allocating real shadow memory for that range which
will increase memory consumption.
You said that you disabled the virtually mapped stacks, which means
that it is not a critical feature for your environment. When is it
supposed to be used? What are the benefits? For now I am leaning
towards automatically disabling virtually mapped stacks when KASAN is
enabled.

Thanks for the report

KASAN vs vmapped stacks

[Andy Lutomirski]

Hi all-

I found two nasty issues with virtually mapped stacks if KASAN is
enabled.  The first issue is a crash: the first non-init stack is
allocated and accessed before KASAN initializes its zero shadow
AFAICT, which means that we switch to that stack and then blow up when
we start recursively faulting on failed accesses to the shadow.

The second issue is that, even if we survive (we initialize the zero
shadow on time), KASAN will fail to protect hte stack.

For now, I just disabled use of virtually mapped stacks if KASAN is
on.  Do you have any easy ideas to fix it?

Thanks,
Andy

KASAN vs vmapped stacks

[Andy Lutomirski]

Hi all-

I found two nasty issues with virtually mapped stacks if KASAN is
enabled.  The first issue is a crash: the first non-init stack is
allocated and accessed before KASAN initializes its zero shadow
AFAICT, which means that we switch to that stack and then blow up when
we start recursively faulting on failed accesses to the shadow.

The second issue is that, even if we survive (we initialize the zero
shadow on time), KASAN will fail to protect hte stack.

For now, I just disabled use of virtually mapped stacks if KASAN is
on.  Do you have any easy ideas to fix it?

Thanks,
Andy