Re: [Xen-devel] [PATCH v2] mm/page_alloc: make bootscrub happen in idle-loop

2018-11-09 Thread Sergey Dyasli 
On 08/11/2018 15:18, Roger Pau Monné wrote:
> On Thu, Nov 08, 2018 at 02:48:40PM +, Sergey Dyasli wrote:
>> (CCing Roger)
>>
>> On 08/11/2018 11:07, Andrew Cooper wrote:
>>> On 08/11/18 10:31, Jan Beulich wrote:
>>> On 07.11.18 at 19:20,  wrote:
> On 09/10/18 16:21, Sergey Dyasli wrote:
>> Scrubbing RAM during boot may take a long time on machines with lots
>> of RAM. Add 'idle' option to bootscrub which marks all pages dirty
>> initially so they will eventually be scrubbed in idle-loop on every
>> online CPU.
>>
>> It's guaranteed that the allocator will return scrubbed pages by doing
>> eager scrubbing during allocation (unless MEMF_no_scrub was provided).
>>
>> Use the new 'idle' option as the default one.
>>
>> Signed-off-by: Sergey Dyasli 
> This patch reliably breaks boot, although its not immediately obvious how:
>
> (d9) (XEN) mcheck_poll: Machine check polling timer started.
> (d9) (XEN) xenoprof: Initialization failed. Intel processor family 6 
> model 
> 60 is not supported
> (d9) (XEN) Dom0 has maximum 400 PIRQs
> (d9) (XEN) [ Xen-4.12-unstable  x86_64  debug=y   Not tainted ]
> (d9) (XEN) CPU:0
> (d9) (XEN) RIP:e008:[] 
> setup.c#cmdline_cook+0x1d/0x77
> (d9) (XEN) RFLAGS: 00010282   CONTEXT: hypervisor
> (d9) (XEN) rax: 82d080406bdc   rbx: 8300c2c2c2c2   rcx: 
> 
> (d9) (XEN) rdx: 0007c7ff   rsi: 8345c24b   rdi: 
> 8345c24b
> (d9) (XEN) rbp: 82d0804b7da8   rsp: 82d0804b7d98   r8:  
> 83003f057000
> (d9) (XEN) r9:  7fff   r10:    r11: 
> 0001
> (d9) (XEN) r12: 83003f0d8100   r13:    r14: 
> 82d0805f33d0
> (d9) (XEN) r15: 0002   cr0: 8005003b   cr4: 
> 001526e0
> (d9) (XEN) cr3: 3fea7000   cr2: 8300c2c2c2c2
> (d9) (XEN) fsb:    gsb:    gss: 
> 
> (d9) (XEN) ds:    es:    fs:    gs:    ss:    cs: e008
> (d9) (XEN) Xen code around  
> (setup.c#cmdline_cook+0x1d/0x77):
> (d9) (XEN)  05 5e fc ff 48 0f 44 d8 <80> 3b 20 75 09 48 83 c3 01 80 3b 20 
> 74 f7 80 3d
> (d9) (XEN) Xen stack trace from rsp=82d0804b7d98:
> [...]
> (d9) (XEN) Xen call trace:
> (d9) (XEN)[] setup.c#cmdline_cook+0x1d/0x77
> (d9) (XEN)[] __start_xen+0x259c/0x292d
> (d9) (XEN)[] __high_start+0x53/0x55
 That's apparently the 2nd cmdline_cook() invocation, when producing
 the Dom0 command line. I would suppose what "loader" points to has
 been scrubbed by the time we get there (with synchronous scrubbing
 APs wouldn't be able to get going with this before reaching
 heap_init_late()).
>>>
>>> This is via a PVH boot (like a lot of my development work), and does
>>> look to be a latent use-after-free.  Dropping the VM down to a single
>>> vcpu causes the problem to go away.
>>>
>>> Sergey is kindly investigating.
>>
>> Yes, this seems to be a bug in Xen PVH boot path. From the serial:
>>
>> (XEN) == mbi->mods_addr 0x46dce0
>>
>> which is marked as usable in e820:
>>
>> (XEN) PVH-e820 RAM map:
>> (XEN)   - 000a (usable)
>> (XEN)  0010 - 4400 (usable)
>> (XEN)  fc00 - fc009040 (ACPI data)
>> (XEN)  feff8000 - feffc000 (reserved)
>> (XEN)  feffc000 - feffd000 (usable)
>> (XEN)  feffd000 - ff00 (reserved)
>>
>> This memory is then given to the allocator and scrubbed by secondary
>> CPUs which leads to use-after-free. Even with fixing the cmdline issue,
>> another FATAL PAGE FAULT occurs further down the boot path:
> 
> Right, shouldn't the scrub be started after Dom0 has been constructed?
> I would say the scrubbing should be started at the same time as
> before, which is just before jumping into Dom0 entry point IIRC?

No, this would only mask the issue again. Although unlikely, that memory
for modules might be given to someone by the allocator, which can lead
to silent memory corruption. Modules are supposed to be freed by
discard_initial_images() which is already called by pvh_load_kernel().

--
Sergey

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH v2] mm/page_alloc: make bootscrub happen in idle-loop

2018-11-08 Thread Roger Pau Monné 
On Thu, Nov 08, 2018 at 02:48:40PM +, Sergey Dyasli wrote:
> (CCing Roger)
> 
> On 08/11/2018 11:07, Andrew Cooper wrote:
> > On 08/11/18 10:31, Jan Beulich wrote:
> > On 07.11.18 at 19:20,  wrote:
> >>> On 09/10/18 16:21, Sergey Dyasli wrote:
>  Scrubbing RAM during boot may take a long time on machines with lots
>  of RAM. Add 'idle' option to bootscrub which marks all pages dirty
>  initially so they will eventually be scrubbed in idle-loop on every
>  online CPU.
> 
>  It's guaranteed that the allocator will return scrubbed pages by doing
>  eager scrubbing during allocation (unless MEMF_no_scrub was provided).
> 
>  Use the new 'idle' option as the default one.
> 
>  Signed-off-by: Sergey Dyasli 
> >>> This patch reliably breaks boot, although its not immediately obvious how:
> >>>
> >>> (d9) (XEN) mcheck_poll: Machine check polling timer started.
> >>> (d9) (XEN) xenoprof: Initialization failed. Intel processor family 6 
> >>> model 
> >>> 60 is not supported
> >>> (d9) (XEN) Dom0 has maximum 400 PIRQs
> >>> (d9) (XEN) [ Xen-4.12-unstable  x86_64  debug=y   Not tainted ]
> >>> (d9) (XEN) CPU:0
> >>> (d9) (XEN) RIP:e008:[] 
> >>> setup.c#cmdline_cook+0x1d/0x77
> >>> (d9) (XEN) RFLAGS: 00010282   CONTEXT: hypervisor
> >>> (d9) (XEN) rax: 82d080406bdc   rbx: 8300c2c2c2c2   rcx: 
> >>> 
> >>> (d9) (XEN) rdx: 0007c7ff   rsi: 8345c24b   rdi: 
> >>> 8345c24b
> >>> (d9) (XEN) rbp: 82d0804b7da8   rsp: 82d0804b7d98   r8:  
> >>> 83003f057000
> >>> (d9) (XEN) r9:  7fff   r10:    r11: 
> >>> 0001
> >>> (d9) (XEN) r12: 83003f0d8100   r13:    r14: 
> >>> 82d0805f33d0
> >>> (d9) (XEN) r15: 0002   cr0: 8005003b   cr4: 
> >>> 001526e0
> >>> (d9) (XEN) cr3: 3fea7000   cr2: 8300c2c2c2c2
> >>> (d9) (XEN) fsb:    gsb:    gss: 
> >>> 
> >>> (d9) (XEN) ds:    es:    fs:    gs:    ss:    cs: e008
> >>> (d9) (XEN) Xen code around  
> >>> (setup.c#cmdline_cook+0x1d/0x77):
> >>> (d9) (XEN)  05 5e fc ff 48 0f 44 d8 <80> 3b 20 75 09 48 83 c3 01 80 3b 20 
> >>> 74 f7 80 3d
> >>> (d9) (XEN) Xen stack trace from rsp=82d0804b7d98:
> >>> [...]
> >>> (d9) (XEN) Xen call trace:
> >>> (d9) (XEN)[] setup.c#cmdline_cook+0x1d/0x77
> >>> (d9) (XEN)[] __start_xen+0x259c/0x292d
> >>> (d9) (XEN)[] __high_start+0x53/0x55
> >> That's apparently the 2nd cmdline_cook() invocation, when producing
> >> the Dom0 command line. I would suppose what "loader" points to has
> >> been scrubbed by the time we get there (with synchronous scrubbing
> >> APs wouldn't be able to get going with this before reaching
> >> heap_init_late()).
> > 
> > This is via a PVH boot (like a lot of my development work), and does
> > look to be a latent use-after-free.  Dropping the VM down to a single
> > vcpu causes the problem to go away.
> > 
> > Sergey is kindly investigating.
> 
> Yes, this seems to be a bug in Xen PVH boot path. From the serial:
> 
> (XEN) == mbi->mods_addr 0x46dce0
> 
> which is marked as usable in e820:
> 
> (XEN) PVH-e820 RAM map:
> (XEN)   - 000a (usable)
> (XEN)  0010 - 4400 (usable)
> (XEN)  fc00 - fc009040 (ACPI data)
> (XEN)  feff8000 - feffc000 (reserved)
> (XEN)  feffc000 - feffd000 (usable)
> (XEN)  feffd000 - ff00 (reserved)
> 
> This memory is then given to the allocator and scrubbed by secondary
> CPUs which leads to use-after-free. Even with fixing the cmdline issue,
> another FATAL PAGE FAULT occurs further down the boot path:

Right, shouldn't the scrub be started after Dom0 has been constructed?
I would say the scrubbing should be started at the same time as
before, which is just before jumping into Dom0 entry point IIRC?

Roger.

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH v2] mm/page_alloc: make bootscrub happen in idle-loop

2018-11-08 Thread Sergey Dyasli 
(CCing Roger)

On 08/11/2018 11:07, Andrew Cooper wrote:
> On 08/11/18 10:31, Jan Beulich wrote:
> On 07.11.18 at 19:20,  wrote:
>>> On 09/10/18 16:21, Sergey Dyasli wrote:
 Scrubbing RAM during boot may take a long time on machines with lots
 of RAM. Add 'idle' option to bootscrub which marks all pages dirty
 initially so they will eventually be scrubbed in idle-loop on every
 online CPU.

 It's guaranteed that the allocator will return scrubbed pages by doing
 eager scrubbing during allocation (unless MEMF_no_scrub was provided).

 Use the new 'idle' option as the default one.

 Signed-off-by: Sergey Dyasli 
>>> This patch reliably breaks boot, although its not immediately obvious how:
>>>
>>> (d9) (XEN) mcheck_poll: Machine check polling timer started.
>>> (d9) (XEN) xenoprof: Initialization failed. Intel processor family 6 model 
>>> 60 is not supported
>>> (d9) (XEN) Dom0 has maximum 400 PIRQs
>>> (d9) (XEN) [ Xen-4.12-unstable  x86_64  debug=y   Not tainted ]
>>> (d9) (XEN) CPU:0
>>> (d9) (XEN) RIP:e008:[] setup.c#cmdline_cook+0x1d/0x77
>>> (d9) (XEN) RFLAGS: 00010282   CONTEXT: hypervisor
>>> (d9) (XEN) rax: 82d080406bdc   rbx: 8300c2c2c2c2   rcx: 
>>> 
>>> (d9) (XEN) rdx: 0007c7ff   rsi: 8345c24b   rdi: 
>>> 8345c24b
>>> (d9) (XEN) rbp: 82d0804b7da8   rsp: 82d0804b7d98   r8:  
>>> 83003f057000
>>> (d9) (XEN) r9:  7fff   r10:    r11: 
>>> 0001
>>> (d9) (XEN) r12: 83003f0d8100   r13:    r14: 
>>> 82d0805f33d0
>>> (d9) (XEN) r15: 0002   cr0: 8005003b   cr4: 
>>> 001526e0
>>> (d9) (XEN) cr3: 3fea7000   cr2: 8300c2c2c2c2
>>> (d9) (XEN) fsb:    gsb:    gss: 
>>> 
>>> (d9) (XEN) ds:    es:    fs:    gs:    ss:    cs: e008
>>> (d9) (XEN) Xen code around  
>>> (setup.c#cmdline_cook+0x1d/0x77):
>>> (d9) (XEN)  05 5e fc ff 48 0f 44 d8 <80> 3b 20 75 09 48 83 c3 01 80 3b 20 
>>> 74 f7 80 3d
>>> (d9) (XEN) Xen stack trace from rsp=82d0804b7d98:
>>> [...]
>>> (d9) (XEN) Xen call trace:
>>> (d9) (XEN)[] setup.c#cmdline_cook+0x1d/0x77
>>> (d9) (XEN)[] __start_xen+0x259c/0x292d
>>> (d9) (XEN)[] __high_start+0x53/0x55
>> That's apparently the 2nd cmdline_cook() invocation, when producing
>> the Dom0 command line. I would suppose what "loader" points to has
>> been scrubbed by the time we get there (with synchronous scrubbing
>> APs wouldn't be able to get going with this before reaching
>> heap_init_late()).
> 
> This is via a PVH boot (like a lot of my development work), and does
> look to be a latent use-after-free.  Dropping the VM down to a single
> vcpu causes the problem to go away.
> 
> Sergey is kindly investigating.

Yes, this seems to be a bug in Xen PVH boot path. From the serial:

(XEN) == mbi->mods_addr 0x46dce0

which is marked as usable in e820:

(XEN) PVH-e820 RAM map:
(XEN)   - 000a (usable)
(XEN)  0010 - 4400 (usable)
(XEN)  fc00 - fc009040 (ACPI data)
(XEN)  feff8000 - feffc000 (reserved)
(XEN)  feffc000 - feffd000 (usable)
(XEN)  feffd000 - ff00 (reserved)

This memory is then given to the allocator and scrubbed by secondary
CPUs which leads to use-after-free. Even with fixing the cmdline issue,
another FATAL PAGE FAULT occurs further down the boot path:

(d16) [183465.829440] (XEN) Xen call trace:
(d16) [183465.829467] (XEN)[] memcmp+0x9/0x3a
(d16) [183465.829494] (XEN)[]
bzimage.c#bzimage_check+0x32/0x71
(d16) [183465.829511] (XEN)[] bzimage_parse+0x22/0xba
(d16) [183465.829528] (XEN)[]
dom0_build.c#pvh_load_kernel+0x82/0x3c0
(d16) [183465.829612] (XEN)[]
dom0_construct_pvh+0x1c9/0x11bf
(d16) [183465.829638] (XEN)[]
construct_dom0+0xd4/0xb0e
(d16) [183465.829655] (XEN)[]
__start_xen+0x2631/0x28b6
(d16) [183465.829682] (XEN)[] __high_start+0x53/0x55
...
(XEN) Faulting linear address: 8f2c2d301202

Looking at mod[0].pa in PVH start info, I suspect that it also gets
overwritten:

(XEN) PVH start info: (pa ffc0)
(XEN)   version:1
(XEN)   flags:  0
(XEN)   nr_modules: 1
(XEN)   modlist_pa: ff70
(XEN)   cmdline_pa: ff90
(XEN)   cmdline:'console=xen,pv dom0=pvh xsm=flask'
(XEN)   rsdp_pa:fc009000
(XEN) mod[0].pa: 005b1000
(XEN) mod[0].size:   04784128
(XEN) mod[0].cmdline_pa: 

The issue is easily reproduced by running Xen as a PVH guest with the
following config:

type="pvh"

vcpus=2
memory=1024
nestedhvm=1

kernel="/root/xen-syms"
ramdisk="/boot/vmlinuz-4.4.0+10"
cmdline="console=xen,pv dom0=pvh xsm=flask"

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org

Re: [Xen-devel] [PATCH v2] mm/page_alloc: make bootscrub happen in idle-loop

2018-11-08 Thread Andrew Cooper 
On 08/11/18 10:31, Jan Beulich wrote:
 On 07.11.18 at 19:20,  wrote:
>> On 09/10/18 16:21, Sergey Dyasli wrote:
>>> Scrubbing RAM during boot may take a long time on machines with lots
>>> of RAM. Add 'idle' option to bootscrub which marks all pages dirty
>>> initially so they will eventually be scrubbed in idle-loop on every
>>> online CPU.
>>>
>>> It's guaranteed that the allocator will return scrubbed pages by doing
>>> eager scrubbing during allocation (unless MEMF_no_scrub was provided).
>>>
>>> Use the new 'idle' option as the default one.
>>>
>>> Signed-off-by: Sergey Dyasli 
>> This patch reliably breaks boot, although its not immediately obvious how:
>>
>> (d9) (XEN) mcheck_poll: Machine check polling timer started.
>> (d9) (XEN) xenoprof: Initialization failed. Intel processor family 6 model 
>> 60 is not supported
>> (d9) (XEN) Dom0 has maximum 400 PIRQs
>> (d9) (XEN) [ Xen-4.12-unstable  x86_64  debug=y   Not tainted ]
>> (d9) (XEN) CPU:0
>> (d9) (XEN) RIP:e008:[] setup.c#cmdline_cook+0x1d/0x77
>> (d9) (XEN) RFLAGS: 00010282   CONTEXT: hypervisor
>> (d9) (XEN) rax: 82d080406bdc   rbx: 8300c2c2c2c2   rcx: 
>> 
>> (d9) (XEN) rdx: 0007c7ff   rsi: 8345c24b   rdi: 
>> 8345c24b
>> (d9) (XEN) rbp: 82d0804b7da8   rsp: 82d0804b7d98   r8:  
>> 83003f057000
>> (d9) (XEN) r9:  7fff   r10:    r11: 
>> 0001
>> (d9) (XEN) r12: 83003f0d8100   r13:    r14: 
>> 82d0805f33d0
>> (d9) (XEN) r15: 0002   cr0: 8005003b   cr4: 
>> 001526e0
>> (d9) (XEN) cr3: 3fea7000   cr2: 8300c2c2c2c2
>> (d9) (XEN) fsb:    gsb:    gss: 
>> 
>> (d9) (XEN) ds:    es:    fs:    gs:    ss:    cs: e008
>> (d9) (XEN) Xen code around  
>> (setup.c#cmdline_cook+0x1d/0x77):
>> (d9) (XEN)  05 5e fc ff 48 0f 44 d8 <80> 3b 20 75 09 48 83 c3 01 80 3b 20 74 
>> f7 80 3d
>> (d9) (XEN) Xen stack trace from rsp=82d0804b7d98:
>> [...]
>> (d9) (XEN) Xen call trace:
>> (d9) (XEN)[] setup.c#cmdline_cook+0x1d/0x77
>> (d9) (XEN)[] __start_xen+0x259c/0x292d
>> (d9) (XEN)[] __high_start+0x53/0x55
> That's apparently the 2nd cmdline_cook() invocation, when producing
> the Dom0 command line. I would suppose what "loader" points to has
> been scrubbed by the time we get there (with synchronous scrubbing
> APs wouldn't be able to get going with this before reaching
> heap_init_late()).

This is via a PVH boot (like a lot of my development work), and does
look to be a latent use-after-free.  Dropping the VM down to a single
vcpu causes the problem to go away.

Sergey is kindly investigating.

~Andrew

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH v2] mm/page_alloc: make bootscrub happen in idle-loop

2018-11-08 Thread Jan Beulich 
>>> On 07.11.18 at 19:20,  wrote:
> On 09/10/18 16:21, Sergey Dyasli wrote:
>> Scrubbing RAM during boot may take a long time on machines with lots
>> of RAM. Add 'idle' option to bootscrub which marks all pages dirty
>> initially so they will eventually be scrubbed in idle-loop on every
>> online CPU.
>>
>> It's guaranteed that the allocator will return scrubbed pages by doing
>> eager scrubbing during allocation (unless MEMF_no_scrub was provided).
>>
>> Use the new 'idle' option as the default one.
>>
>> Signed-off-by: Sergey Dyasli 
> 
> This patch reliably breaks boot, although its not immediately obvious how:
> 
> (d9) (XEN) mcheck_poll: Machine check polling timer started.
> (d9) (XEN) xenoprof: Initialization failed. Intel processor family 6 model 
> 60 is not supported
> (d9) (XEN) Dom0 has maximum 400 PIRQs
> (d9) (XEN) [ Xen-4.12-unstable  x86_64  debug=y   Not tainted ]
> (d9) (XEN) CPU:0
> (d9) (XEN) RIP:e008:[] setup.c#cmdline_cook+0x1d/0x77
> (d9) (XEN) RFLAGS: 00010282   CONTEXT: hypervisor
> (d9) (XEN) rax: 82d080406bdc   rbx: 8300c2c2c2c2   rcx: 
> 
> (d9) (XEN) rdx: 0007c7ff   rsi: 8345c24b   rdi: 
> 8345c24b
> (d9) (XEN) rbp: 82d0804b7da8   rsp: 82d0804b7d98   r8:  
> 83003f057000
> (d9) (XEN) r9:  7fff   r10:    r11: 
> 0001
> (d9) (XEN) r12: 83003f0d8100   r13:    r14: 
> 82d0805f33d0
> (d9) (XEN) r15: 0002   cr0: 8005003b   cr4: 
> 001526e0
> (d9) (XEN) cr3: 3fea7000   cr2: 8300c2c2c2c2
> (d9) (XEN) fsb:    gsb:    gss: 
> 
> (d9) (XEN) ds:    es:    fs:    gs:    ss:    cs: e008
> (d9) (XEN) Xen code around  
> (setup.c#cmdline_cook+0x1d/0x77):
> (d9) (XEN)  05 5e fc ff 48 0f 44 d8 <80> 3b 20 75 09 48 83 c3 01 80 3b 20 74 
> f7 80 3d
> (d9) (XEN) Xen stack trace from rsp=82d0804b7d98:
>[...]
> (d9) (XEN) Xen call trace:
> (d9) (XEN)[] setup.c#cmdline_cook+0x1d/0x77
> (d9) (XEN)[] __start_xen+0x259c/0x292d
> (d9) (XEN)[] __high_start+0x53/0x55

That's apparently the 2nd cmdline_cook() invocation, when producing
the Dom0 command line. I would suppose what "loader" points to has
been scrubbed by the time we get there (with synchronous scrubbing
APs wouldn't be able to get going with this before reaching
heap_init_late()).

Jan



___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH v2] mm/page_alloc: make bootscrub happen in idle-loop

2018-11-08 Thread Sergey Dyasli 
On 07/11/2018 18:20, Andrew Cooper wrote:
> On 09/10/18 16:21, Sergey Dyasli wrote:
>> Scrubbing RAM during boot may take a long time on machines with lots
>> of RAM. Add 'idle' option to bootscrub which marks all pages dirty
>> initially so they will eventually be scrubbed in idle-loop on every
>> online CPU.
>>
>> It's guaranteed that the allocator will return scrubbed pages by doing
>> eager scrubbing during allocation (unless MEMF_no_scrub was provided).
>>
>> Use the new 'idle' option as the default one.
>>
>> Signed-off-by: Sergey Dyasli 
> 
> This patch reliably breaks boot, although its not immediately obvious how:
> 
> (d9) (XEN) mcheck_poll: Machine check polling timer started.
> (d9) (XEN) xenoprof: Initialization failed. Intel processor family 6 model 60 
> is not supported
> (d9) (XEN) Dom0 has maximum 400 PIRQs
> (d9) (XEN) [ Xen-4.12-unstable  x86_64  debug=y   Not tainted ]
> (d9) (XEN) CPU:0
> (d9) (XEN) RIP:e008:[] setup.c#cmdline_cook+0x1d/0x77
> (d9) (XEN) RFLAGS: 00010282   CONTEXT: hypervisor
> (d9) (XEN) rax: 82d080406bdc   rbx: 8300c2c2c2c2   rcx: 
> 
> (d9) (XEN) rdx: 0007c7ff   rsi: 8345c24b   rdi: 
> 8345c24b
> (d9) (XEN) rbp: 82d0804b7da8   rsp: 82d0804b7d98   r8:  
> 83003f057000
> (d9) (XEN) r9:  7fff   r10:    r11: 
> 0001
> (d9) (XEN) r12: 83003f0d8100   r13:    r14: 
> 82d0805f33d0
> (d9) (XEN) r15: 0002   cr0: 8005003b   cr4: 
> 001526e0
> (d9) (XEN) cr3: 3fea7000   cr2: 8300c2c2c2c2
> (d9) (XEN) fsb:    gsb:    gss: 
> 
> (d9) (XEN) ds:    es:    fs:    gs:    ss:    cs: e008
> (d9) (XEN) Xen code around  
> (setup.c#cmdline_cook+0x1d/0x77):
> (d9) (XEN)  05 5e fc ff 48 0f 44 d8 <80> 3b 20 75 09 48 83 c3 01 80 3b 20 74 
> f7 80 3d
> (d9) (XEN) Xen stack trace from rsp=82d0804b7d98:
> (d9) (XEN) 8300c2c2c2c2 82d0804b7ee8 
> 82d080443b7f
> (d9) (XEN) 003f3480 0002 
> 0002
> (d9) (XEN)0002 0002 0002 
> 0001
> (d9) (XEN)0001 0003 000feffc 
> 
> (d9) (XEN)000feffd  00800163 
> feffd000
> (d9) (XEN)8345c24b 0002 0001 
> 0001
> (d9) (XEN)8348da80 82d08048db00  
> 
> (d9) (XEN) 00020004 0040 
> 0400
> (d9) (XEN)0008 0001006e 0003 
> 02f8
> (d9) (XEN)   
> 
> (d9) (XEN)   
> 82d0802000f3
> (d9) (XEN)   
> 
> (d9) (XEN)   
> 
> (d9) (XEN)   
> 
> (d9) (XEN)   
> 
> (d9) (XEN)   
> 
> (d9) (XEN)   
> 
> (d9) (XEN)  83003f0ce000 
> 
> (d9) (XEN)001526e0   
> 0600
> (d9) (XEN)0018
> (d9) (XEN) Xen call trace:
> (d9) (XEN)[] setup.c#cmdline_cook+0x1d/0x77
> (d9) (XEN)[] __start_xen+0x259c/0x292d
> (d9) (XEN)[] __high_start+0x53/0x55
> (d9) (XEN) 
> (d9) (XEN) Pagetable walk from 8300c2c2c2c2:
> (d9) (XEN)  L4[0x106] = 80003fea5063 
> (d9) (XEN)  L3[0x003] = 3fea2063 
> (d9) (XEN)  L2[0x016] =  
> (d9) (XEN) 
> (d9) (XEN) 
> (d9) (XEN) Panic on CPU 0:
> (d9) (XEN) FATAL PAGE FAULT
> (d9) (XEN) [error_code=]
> (d9) (XEN) Faulting linear address: 8300c2c2c2c2
> (d9) (XEN) 
> (d9) (XEN) 
> (d9) (XEN) Reboot in five seconds...
> 
> The low part of 0x8300c2c2c2c2 looks to be poisoned, so
> __va(mod[0].string) is obviously turning out to be junk.

0xc2 is a SCRUB_PATTERN, so my patch might have uncovered a real issue.
There are 2 implications of idle scrub:

1. alloc_xenheap_pages() might return scrubbed memory (despite
   passing MEMF_no_scrub, and after secondary CPUs enter idle-loop)

2. alloc_domheap_pages() will return scrubbed memory by default
   during Xen boot

What is the exact place of this crash? Maybe zeroing of allocated pages
is needed there? Can you

Re: [Xen-devel] [PATCH v2] mm/page_alloc: make bootscrub happen in idle-loop

2018-11-07 Thread Andrew Cooper 
On 09/10/18 16:21, Sergey Dyasli wrote:
> Scrubbing RAM during boot may take a long time on machines with lots
> of RAM. Add 'idle' option to bootscrub which marks all pages dirty
> initially so they will eventually be scrubbed in idle-loop on every
> online CPU.
>
> It's guaranteed that the allocator will return scrubbed pages by doing
> eager scrubbing during allocation (unless MEMF_no_scrub was provided).
>
> Use the new 'idle' option as the default one.
>
> Signed-off-by: Sergey Dyasli 

This patch reliably breaks boot, although its not immediately obvious how:

(d9) (XEN) mcheck_poll: Machine check polling timer started.
(d9) (XEN) xenoprof: Initialization failed. Intel processor family 6 model 60 
is not supported
(d9) (XEN) Dom0 has maximum 400 PIRQs
(d9) (XEN) [ Xen-4.12-unstable  x86_64  debug=y   Not tainted ]
(d9) (XEN) CPU:0
(d9) (XEN) RIP:e008:[] setup.c#cmdline_cook+0x1d/0x77
(d9) (XEN) RFLAGS: 00010282   CONTEXT: hypervisor
(d9) (XEN) rax: 82d080406bdc   rbx: 8300c2c2c2c2   rcx: 
(d9) (XEN) rdx: 0007c7ff   rsi: 8345c24b   rdi: 8345c24b
(d9) (XEN) rbp: 82d0804b7da8   rsp: 82d0804b7d98   r8:  83003f057000
(d9) (XEN) r9:  7fff   r10:    r11: 0001
(d9) (XEN) r12: 83003f0d8100   r13:    r14: 82d0805f33d0
(d9) (XEN) r15: 0002   cr0: 8005003b   cr4: 001526e0
(d9) (XEN) cr3: 3fea7000   cr2: 8300c2c2c2c2
(d9) (XEN) fsb:    gsb:    gss: 
(d9) (XEN) ds:    es:    fs:    gs:    ss:    cs: e008
(d9) (XEN) Xen code around  (setup.c#cmdline_cook+0x1d/0x77):
(d9) (XEN)  05 5e fc ff 48 0f 44 d8 <80> 3b 20 75 09 48 83 c3 01 80 3b 20 74 f7 
80 3d
(d9) (XEN) Xen stack trace from rsp=82d0804b7d98:
(d9) (XEN) 8300c2c2c2c2 82d0804b7ee8 
82d080443b7f
(d9) (XEN) 003f3480 0002 
0002
(d9) (XEN)0002 0002 0002 
0001
(d9) (XEN)0001 0003 000feffc 

(d9) (XEN)000feffd  00800163 
feffd000
(d9) (XEN)8345c24b 0002 0001 
0001
(d9) (XEN)8348da80 82d08048db00  

(d9) (XEN) 00020004 0040 
0400
(d9) (XEN)0008 0001006e 0003 
02f8
(d9) (XEN)   

(d9) (XEN)   
82d0802000f3
(d9) (XEN)   

(d9) (XEN)   

(d9) (XEN)   

(d9) (XEN)   

(d9) (XEN)   

(d9) (XEN)   

(d9) (XEN)  83003f0ce000 

(d9) (XEN)001526e0   
0600
(d9) (XEN)0018
(d9) (XEN) Xen call trace:
(d9) (XEN)[] setup.c#cmdline_cook+0x1d/0x77
(d9) (XEN)[] __start_xen+0x259c/0x292d
(d9) (XEN)[] __high_start+0x53/0x55
(d9) (XEN) 
(d9) (XEN) Pagetable walk from 8300c2c2c2c2:
(d9) (XEN)  L4[0x106] = 80003fea5063 
(d9) (XEN)  L3[0x003] = 3fea2063 
(d9) (XEN)  L2[0x016] =  
(d9) (XEN) 
(d9) (XEN) 
(d9) (XEN) Panic on CPU 0:
(d9) (XEN) FATAL PAGE FAULT
(d9) (XEN) [error_code=]
(d9) (XEN) Faulting linear address: 8300c2c2c2c2
(d9) (XEN) 
(d9) (XEN) 
(d9) (XEN) Reboot in five seconds...

The low part of 0x8300c2c2c2c2 looks to be poisoned, so
__va(mod[0].string) is obviously turning out to be junk.

~Andrew

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH v2] mm/page_alloc: make bootscrub happen in idle-loop

2018-10-15 Thread Sergey Dyasli 
On 12/10/18 14:40, Jan Beulich wrote:
 On 09.10.18 at 17:21,  wrote:
>> --- a/xen/common/page_alloc.c
>> +++ b/xen/common/page_alloc.c
>> @@ -161,8 +161,42 @@ string_param("badpage", opt_badpage);
>>  /*
>>   * no-bootscrub -> Free pages are not zeroed during boot.
>>   */
>> -static bool_t opt_bootscrub __initdata = 1;
>> -boolean_param("bootscrub", opt_bootscrub);
>> +enum bootscrub_mode {
>> +BOOTSCRUB_OFF = 0,
> 
> The "= 0" is pointless.

I don't mind this change.

>> @@ -2039,8 +2077,24 @@ void __init heap_init_late(void)
>>   */
>>  setup_low_mem_virq();
>>  
>> -if ( opt_bootscrub )
>> +switch ( opt_bootscrub )
>> +{
>> +default:
>> +ASSERT_UNREACHABLE();
>> +/* Fall through */
>> +
>> +case BOOTSCRUB_IDLE:
>> +printk("Scrubbing Free RAM on %d nodes in background\n",
>> +   num_online_nodes());
> 
> Still the question whether this printk(), and in particular the inclusion
> of the node count, is meaningful in any way. Other than this

I don't have any strong opinion about how this printk() statement should
look like. It can be changed to whatever maintainers find more appropriate.

> Reviewed-by: Jan Beulich 
> and one or both changes would be easy enough to make while
> committing, provided we can reach agreement.

Thanks,
Sergey

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH v2] mm/page_alloc: make bootscrub happen in idle-loop

2018-10-12 Thread Jan Beulich 
>>> On 09.10.18 at 17:21,  wrote:
> --- a/xen/common/page_alloc.c
> +++ b/xen/common/page_alloc.c
> @@ -161,8 +161,42 @@ string_param("badpage", opt_badpage);
>  /*
>   * no-bootscrub -> Free pages are not zeroed during boot.
>   */
> -static bool_t opt_bootscrub __initdata = 1;
> -boolean_param("bootscrub", opt_bootscrub);
> +enum bootscrub_mode {
> +BOOTSCRUB_OFF = 0,

The "= 0" is pointless.

> @@ -2039,8 +2077,24 @@ void __init heap_init_late(void)
>   */
>  setup_low_mem_virq();
>  
> -if ( opt_bootscrub )
> +switch ( opt_bootscrub )
> +{
> +default:
> +ASSERT_UNREACHABLE();
> +/* Fall through */
> +
> +case BOOTSCRUB_IDLE:
> +printk("Scrubbing Free RAM on %d nodes in background\n",
> +   num_online_nodes());

Still the question whether this printk(), and in particular the inclusion
of the node count, is meaningful in any way. Other than this
Reviewed-by: Jan Beulich 
and one or both changes would be easy enough to make while
committing, provided we can reach agreement.

Jan



___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[Xen-devel] [PATCH v2] mm/page_alloc: make bootscrub happen in idle-loop

2018-10-09 Thread Sergey Dyasli 
Scrubbing RAM during boot may take a long time on machines with lots
of RAM. Add 'idle' option to bootscrub which marks all pages dirty
initially so they will eventually be scrubbed in idle-loop on every
online CPU.

It's guaranteed that the allocator will return scrubbed pages by doing
eager scrubbing during allocation (unless MEMF_no_scrub was provided).

Use the new 'idle' option as the default one.

Signed-off-by: Sergey Dyasli 
---
v1 --> v2:
- dropped comment about performance
- changed default to 'idle'
- changed type of opt_bootscrub to enum
- restored __initdata for opt_bootscrub
- call parse_bool() first during parsing
- using switch() in heap_init_late()

Note: The message "Scrubbing Free RAM on %d nodes" corresponds to
the similar one in scrub_heap_pages()

CC: Andrew Cooper 
CC: Boris Ostrovsky 
CC: George Dunlap 
CC: Jan Beulich 
CC: Julien Grall 
CC: Wei Liu 
---
 docs/misc/xen-command-line.markdown |  9 -
 xen/common/page_alloc.c | 62 +++--
 2 files changed, 65 insertions(+), 6 deletions(-)

diff --git a/docs/misc/xen-command-line.markdown 
b/docs/misc/xen-command-line.markdown
index 1ffd586224..9c15d52bf7 100644
--- a/docs/misc/xen-command-line.markdown
+++ b/docs/misc/xen-command-line.markdown
@@ -227,14 +227,19 @@ that byte `0x12345678` is bad, you would place 
`badpage=0x12345` on
 Xen's command line.
 
 ### bootscrub
-> `= `
+> `= idle | `
 
-> Default: `true`
+> Default: `idle`
 
 Scrub free RAM during boot.  This is a safety feature to prevent
 accidentally leaking sensitive VM data into other VMs if Xen crashes
 and reboots.
 
+In `idle` mode, RAM is scrubbed in background on all CPUs during idle-loop
+with a guarantee that memory allocations always provide scrubbed pages.
+This option reduces boot time on machines with a large amount of RAM while
+still providing security benefits.
+
 ### bootscrub\_chunk
 > `= `
 
diff --git a/xen/common/page_alloc.c b/xen/common/page_alloc.c
index 16e1b0c357..7df5d5c545 100644
--- a/xen/common/page_alloc.c
+++ b/xen/common/page_alloc.c
@@ -161,8 +161,42 @@ string_param("badpage", opt_badpage);
 /*
  * no-bootscrub -> Free pages are not zeroed during boot.
  */
-static bool_t opt_bootscrub __initdata = 1;
-boolean_param("bootscrub", opt_bootscrub);
+enum bootscrub_mode {
+BOOTSCRUB_OFF = 0,
+BOOTSCRUB_ON,
+BOOTSCRUB_IDLE,
+};
+static enum bootscrub_mode __initdata opt_bootscrub = BOOTSCRUB_IDLE;
+static int __init parse_bootscrub_param(const char *s)
+{
+/* Interpret 'bootscrub' alone in its positive boolean form */
+if ( *s == '\0' )
+{
+opt_bootscrub = BOOTSCRUB_ON;
+return 0;
+}
+
+switch ( parse_bool(s, NULL) )
+{
+case 0:
+opt_bootscrub = BOOTSCRUB_OFF;
+break;
+
+case 1:
+opt_bootscrub = BOOTSCRUB_ON;
+break;
+
+default:
+if ( !strcmp(s, "idle") )
+opt_bootscrub = BOOTSCRUB_IDLE;
+else
+return -EINVAL;
+break;
+}
+
+return 0;
+}
+custom_param("bootscrub", parse_bootscrub_param);
 
 /*
  * bootscrub_chunk -> Amount of bytes to scrub lockstep on non-SMT CPUs
@@ -1726,6 +1760,7 @@ static void init_heap_pages(
 struct page_info *pg, unsigned long nr_pages)
 {
 unsigned long i;
+bool idle_scrub = false;
 
 /*
  * Some pages may not go through the boot allocator (e.g reserved
@@ -1737,6 +1772,9 @@ static void init_heap_pages(
 first_valid_mfn = mfn_min(page_to_mfn(pg), first_valid_mfn);
 spin_unlock(_lock);
 
+if ( system_state < SYS_STATE_active && opt_bootscrub == BOOTSCRUB_IDLE )
+idle_scrub = true;
+
 for ( i = 0; i < nr_pages; i++ )
 {
 unsigned int nid = phys_to_nid(page_to_maddr(pg+i));
@@ -1763,7 +1801,7 @@ static void init_heap_pages(
 nr_pages -= n;
 }
 
-free_heap_pages(pg + i, 0, scrub_debug);
+free_heap_pages(pg + i, 0, scrub_debug || idle_scrub);
 }
 }
 
@@ -2039,8 +2077,24 @@ void __init heap_init_late(void)
  */
 setup_low_mem_virq();
 
-if ( opt_bootscrub )
+switch ( opt_bootscrub )
+{
+default:
+ASSERT_UNREACHABLE();
+/* Fall through */
+
+case BOOTSCRUB_IDLE:
+printk("Scrubbing Free RAM on %d nodes in background\n",
+   num_online_nodes());
+break;
+
+case BOOTSCRUB_ON:
 scrub_heap_pages();
+break;
+
+case BOOTSCRUB_OFF:
+break;
+}
 }
 
 
-- 
2.17.1


___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel