Re: [Xen-devel] [PATCH RFC] mm/memory_hotplug: Introduce memory block types

2018-11-27 Thread David Hildenbrand
On 27.11.18 17:32, Michal Suchánek wrote: > On Mon, 26 Nov 2018 16:59:14 +0100 > David Hildenbrand wrote: > >> On 26.11.18 15:20, Michal Suchánek wrote: >>> On Mon, 26 Nov 2018 14:33:29 +0100 >>> David Hildenbrand wrote: >>> On 26.11.18 13:30, David Hildenbrand wrote: > On

Re: [Xen-devel] [PATCH RFC] mm/memory_hotplug: Introduce memory block types

2018-11-27 Thread Michal Suchánek
On Mon, 26 Nov 2018 16:59:14 +0100 David Hildenbrand wrote: > On 26.11.18 15:20, Michal Suchánek wrote: > > On Mon, 26 Nov 2018 14:33:29 +0100 > > David Hildenbrand wrote: > > > >> On 26.11.18 13:30, David Hildenbrand wrote: > >>> On 23.11.18 19:06, Michal Suchánek wrote: > > >

Re: [Xen-devel] [PATCH RFC] mm/memory_hotplug: Introduce memory block types

2018-11-26 Thread David Hildenbrand
On 26.11.18 15:20, Michal Suchánek wrote: > On Mon, 26 Nov 2018 14:33:29 +0100 > David Hildenbrand wrote: > >> On 26.11.18 13:30, David Hildenbrand wrote: >>> On 23.11.18 19:06, Michal Suchánek wrote: > If we are going to fake the driver information we may as well add the type

Re: [Xen-devel] [PATCH RFC] mm/memory_hotplug: Introduce memory block types

2018-11-26 Thread Michal Suchánek
On Mon, 26 Nov 2018 14:33:29 +0100 David Hildenbrand wrote: > On 26.11.18 13:30, David Hildenbrand wrote: > > On 23.11.18 19:06, Michal Suchánek wrote: > >> > >> If we are going to fake the driver information we may as well add the > >> type attribute and be done with it. > >> > >> I think

Re: [Xen-devel] [PATCH RFC] mm/memory_hotplug: Introduce memory block types

2018-11-26 Thread David Hildenbrand
On 26.11.18 13:30, David Hildenbrand wrote: > On 23.11.18 19:06, Michal Suchánek wrote: >> On Fri, 23 Nov 2018 12:13:58 +0100 >> David Hildenbrand wrote: >> >>> On 28.09.18 17:03, David Hildenbrand wrote: How to/when to online hotplugged memory is hard to manage for distributions

Re: [Xen-devel] [PATCH RFC] mm/memory_hotplug: Introduce memory block types

2018-11-26 Thread David Hildenbrand
On 23.11.18 19:06, Michal Suchánek wrote: > On Fri, 23 Nov 2018 12:13:58 +0100 > David Hildenbrand wrote: > >> On 28.09.18 17:03, David Hildenbrand wrote: >>> How to/when to online hotplugged memory is hard to manage for >>> distributions because different memory types are to be treated

Re: [Xen-devel] [PATCH RFC] mm/memory_hotplug: Introduce memory block types

2018-11-23 Thread Michal Suchánek
On Fri, 23 Nov 2018 12:13:58 +0100 David Hildenbrand wrote: > On 28.09.18 17:03, David Hildenbrand wrote: > > How to/when to online hotplugged memory is hard to manage for > > distributions because different memory types are to be treated differently. > > Right now, we need complicated udev

Re: [Xen-devel] [PATCH RFC] mm/memory_hotplug: Introduce memory block types

2018-11-23 Thread David Hildenbrand
On 28.09.18 17:03, David Hildenbrand wrote: > How to/when to online hotplugged memory is hard to manage for > distributions because different memory types are to be treated differently. > Right now, we need complicated udev rules that e.g. check if we are > running on s390x, on a physical system

Re: [Xen-devel] [PATCH RFC] mm/memory_hotplug: Introduce memory block types

2018-10-05 Thread David Hildenbrand
On 04/10/2018 19:50, Michal Suchánek wrote: > On Thu, 4 Oct 2018 17:45:13 +0200 > David Hildenbrand wrote: > >> On 04/10/2018 17:28, Michal Suchánek wrote: > >>> >>> The state of the art is to determine what to do with hotplugged >>> memory in userspace based on platform and virtualization

Re: [Xen-devel] [PATCH RFC] mm/memory_hotplug: Introduce memory block types

2018-10-04 Thread Michal Suchánek
On Thu, 4 Oct 2018 17:45:13 +0200 David Hildenbrand wrote: > On 04/10/2018 17:28, Michal Suchánek wrote: > > > > The state of the art is to determine what to do with hotplugged > > memory in userspace based on platform and virtualization type. > > Exactly. > > > > > Changing the default

Re: [Xen-devel] [PATCH RFC] mm/memory_hotplug: Introduce memory block types

2018-10-04 Thread David Hildenbrand
On 04/10/2018 17:28, Michal Suchánek wrote: > On Thu, 4 Oct 2018 10:13:48 +0200 > David Hildenbrand wrote: > > ok, so what is the problem here? > > Handling the hotplug in userspace through udev may be suboptimal and > kernel handling might be faster but that's orthogonal to the problem at >

Re: [Xen-devel] [PATCH RFC] mm/memory_hotplug: Introduce memory block types

2018-10-04 Thread Michal Suchánek
On Thu, 4 Oct 2018 10:13:48 +0200 David Hildenbrand wrote: ok, so what is the problem here? Handling the hotplug in userspace through udev may be suboptimal and kernel handling might be faster but that's orthogonal to the problem at hand. The state of the art is to determine what to do with

Re: [Xen-devel] [PATCH RFC] mm/memory_hotplug: Introduce memory block types

2018-10-04 Thread David Hildenbrand
On 04/10/2018 08:19, Michal Hocko wrote: > On Wed 03-10-18 19:14:05, David Hildenbrand wrote: >> On 03/10/2018 16:34, Vitaly Kuznetsov wrote: >>> Dave Hansen writes: >>> On 10/03/2018 06:52 AM, Vitaly Kuznetsov wrote: > It is more than just memmaps (e.g. forking udev process doing memory

Re: [Xen-devel] [PATCH RFC] mm/memory_hotplug: Introduce memory block types

2018-10-04 Thread David Hildenbrand
On 03/10/2018 16:24, Michal Hocko wrote: > On Wed 03-10-18 15:52:24, Vitaly Kuznetsov wrote: > [...] >>> As David said some of the memory cannot be onlined without further steps >>> (e.g. when it is standby as David called it) and then I fail to see how >>> eBPF help in any way. >> >> and also, we

Re: [Xen-devel] [PATCH RFC] mm/memory_hotplug: Introduce memory block types

2018-10-04 Thread David Hildenbrand
On 01/10/2018 18:24, Dave Hansen wrote: >> How should a policy in user space look like when new memory gets added >> - on s390x? Not onlining paravirtualized memory is very wrong. > > Because we're going to balloon it away in a moment anyway? No, rether somebody wanted this VM to have more

Re: [Xen-devel] [PATCH RFC] mm/memory_hotplug: Introduce memory block types

2018-10-04 Thread David Hildenbrand
On 04/10/2018 08:28, Michal Hocko wrote: > On Wed 03-10-18 19:00:29, David Hildenbrand wrote: > [...] >> Let me rephrase: You state that user space has to make the decision and >> that user should be able to set/reconfigure rules. That is perfectly fine. >> >> But then we should give user space

Re: [Xen-devel] [PATCH RFC] mm/memory_hotplug: Introduce memory block types

2018-10-04 Thread Michal Hocko
On Wed 03-10-18 19:00:29, David Hildenbrand wrote: [...] > Let me rephrase: You state that user space has to make the decision and > that user should be able to set/reconfigure rules. That is perfectly fine. > > But then we should give user space access to sufficient information to > make a

Re: [Xen-devel] [PATCH RFC] mm/memory_hotplug: Introduce memory block types

2018-10-04 Thread Michal Hocko
On Wed 03-10-18 19:14:05, David Hildenbrand wrote: > On 03/10/2018 16:34, Vitaly Kuznetsov wrote: > > Dave Hansen writes: > > > >> On 10/03/2018 06:52 AM, Vitaly Kuznetsov wrote: > >>> It is more than just memmaps (e.g. forking udev process doing memory > >>> onlining also needs memory) but yes,

Re: [Xen-devel] [PATCH RFC] mm/memory_hotplug: Introduce memory block types

2018-10-03 Thread David Hildenbrand
On 03/10/2018 16:34, Vitaly Kuznetsov wrote: > Dave Hansen writes: > >> On 10/03/2018 06:52 AM, Vitaly Kuznetsov wrote: >>> It is more than just memmaps (e.g. forking udev process doing memory >>> onlining also needs memory) but yes, the main idea is to make the >>> onlining synchronous with

Re: [Xen-devel] [PATCH RFC] mm/memory_hotplug: Introduce memory block types

2018-10-03 Thread David Hildenbrand
On 03/10/2018 16:24, Michal Hocko wrote: > On Wed 03-10-18 15:52:24, Vitaly Kuznetsov wrote: > [...] >>> As David said some of the memory cannot be onlined without further steps >>> (e.g. when it is standby as David called it) and then I fail to see how >>> eBPF help in any way. >> >> and also, we

Re: [Xen-devel] [PATCH RFC] mm/memory_hotplug: Introduce memory block types

2018-10-03 Thread David Hildenbrand
On 03/10/2018 15:54, Michal Hocko wrote: > On Tue 02-10-18 17:25:19, David Hildenbrand wrote: >> On 02/10/2018 15:47, Michal Hocko wrote: > [...] >>> Zone imbalance is an inherent problem of the highmem zone. It is >>> essentially the highmem zone we all loved so much back in 32b days. >>> Yes the

Re: [Xen-devel] [PATCH RFC] mm/memory_hotplug: Introduce memory block types

2018-10-03 Thread Vitaly Kuznetsov
Dave Hansen writes: > On 10/03/2018 06:52 AM, Vitaly Kuznetsov wrote: >> It is more than just memmaps (e.g. forking udev process doing memory >> onlining also needs memory) but yes, the main idea is to make the >> onlining synchronous with hotplug. > > That's a good theoretical concern. > > But,

Re: [Xen-devel] [PATCH RFC] mm/memory_hotplug: Introduce memory block types

2018-10-03 Thread Michal Hocko
On Wed 03-10-18 15:52:24, Vitaly Kuznetsov wrote: [...] > > As David said some of the memory cannot be onlined without further steps > > (e.g. when it is standby as David called it) and then I fail to see how > > eBPF help in any way. > > and also, we can fight till the end of days here trying to

Re: [Xen-devel] [PATCH RFC] mm/memory_hotplug: Introduce memory block types

2018-10-03 Thread Dave Hansen
On 10/03/2018 06:52 AM, Vitaly Kuznetsov wrote: > It is more than just memmaps (e.g. forking udev process doing memory > onlining also needs memory) but yes, the main idea is to make the > onlining synchronous with hotplug. That's a good theoretical concern. But, is it a problem we need to solve

Re: [Xen-devel] [PATCH RFC] mm/memory_hotplug: Introduce memory block types

2018-10-03 Thread Michal Hocko
On Tue 02-10-18 17:25:19, David Hildenbrand wrote: > On 02/10/2018 15:47, Michal Hocko wrote: [...] > > Zone imbalance is an inherent problem of the highmem zone. It is > > essentially the highmem zone we all loved so much back in 32b days. > > Yes the movable zone doesn't have any addressing

Re: [Xen-devel] [PATCH RFC] mm/memory_hotplug: Introduce memory block types

2018-10-03 Thread Vitaly Kuznetsov
Michal Hocko writes: > On Wed 03-10-18 15:38:04, Vitaly Kuznetsov wrote: >> David Hildenbrand writes: >> >> > On 02/10/2018 15:47, Michal Hocko wrote: >> ... >> >> >> >> Why do you need a generic hotplug rule in the first place? Why don't you >> >> simply provide different set of rules for

Re: [Xen-devel] [PATCH RFC] mm/memory_hotplug: Introduce memory block types

2018-10-03 Thread Michal Hocko
On Wed 03-10-18 15:38:04, Vitaly Kuznetsov wrote: > David Hildenbrand writes: > > > On 02/10/2018 15:47, Michal Hocko wrote: > ... > >> > >> Why do you need a generic hotplug rule in the first place? Why don't you > >> simply provide different set of rules for different usecases? Let users > >>

Re: [Xen-devel] [PATCH RFC] mm/memory_hotplug: Introduce memory block types

2018-10-03 Thread Vitaly Kuznetsov
David Hildenbrand writes: > On 02/10/2018 15:47, Michal Hocko wrote: ... >> >> Why do you need a generic hotplug rule in the first place? Why don't you >> simply provide different set of rules for different usecases? Let users >> decide which usecase they prefer rather than try to be clever

Re: [Xen-devel] [PATCH RFC] mm/memory_hotplug: Introduce memory block types

2018-10-02 Thread David Hildenbrand
On 02/10/2018 15:47, Michal Hocko wrote: > On Mon 01-10-18 11:34:25, David Hildenbrand wrote: >> On 01/10/2018 10:40, Michal Hocko wrote: >>> On Fri 28-09-18 17:03:57, David Hildenbrand wrote: >>> [...] >>> >>> I haven't read the patch itself but I just wanted to note one thing >>> about this part

Re: [Xen-devel] [PATCH RFC] mm/memory_hotplug: Introduce memory block types

2018-10-02 Thread Michal Hocko
On Mon 01-10-18 11:34:25, David Hildenbrand wrote: > On 01/10/2018 10:40, Michal Hocko wrote: > > On Fri 28-09-18 17:03:57, David Hildenbrand wrote: > > [...] > > > > I haven't read the patch itself but I just wanted to note one thing > > about this part > > > >> For paravirtualized devices it

Re: [Xen-devel] [PATCH RFC] mm/memory_hotplug: Introduce memory block types

2018-10-01 Thread Dave Hansen
> How should a policy in user space look like when new memory gets added > - on s390x? Not onlining paravirtualized memory is very wrong. Because we're going to balloon it away in a moment anyway? We have auto-onlining. Why isn't that being used on s390? > So the type of memory is very

Re: [Xen-devel] [PATCH RFC] mm/memory_hotplug: Introduce memory block types

2018-10-01 Thread David Hildenbrand
On 01/10/2018 10:40, Michal Hocko wrote: > On Fri 28-09-18 17:03:57, David Hildenbrand wrote: > [...] > > I haven't read the patch itself but I just wanted to note one thing > about this part > >> For paravirtualized devices it is relevant that memory is onlined as >> quickly as possible after

Re: [Xen-devel] [PATCH RFC] mm/memory_hotplug: Introduce memory block types

2018-10-01 Thread David Hildenbrand
On 28/09/2018 19:02, Dave Hansen wrote: > It's really nice if these kinds of things are broken up. First, replace > the old want_memblock parameter, then add the parameter to the > __add_page() calls. Definitely, once we agree that is is not nuts, I will split it up for the next version :) >

Re: [Xen-devel] [PATCH RFC] mm/memory_hotplug: Introduce memory block types

2018-10-01 Thread Michal Hocko
On Fri 28-09-18 17:03:57, David Hildenbrand wrote: [...] I haven't read the patch itself but I just wanted to note one thing about this part > For paravirtualized devices it is relevant that memory is onlined as > quickly as possible after adding - and that it is added to the NORMAL > zone.

Re: [Xen-devel] [PATCH RFC] mm/memory_hotplug: Introduce memory block types

2018-09-28 Thread Dave Hansen
It's really nice if these kinds of things are broken up. First, replace the old want_memblock parameter, then add the parameter to the __add_page() calls. > +/* > + * NONE: No memory block is to be created (e.g. device memory). > + * NORMAL: Memory block that represents normal (boot or

[Xen-devel] [PATCH RFC] mm/memory_hotplug: Introduce memory block types

2018-09-28 Thread David Hildenbrand
How to/when to online hotplugged memory is hard to manage for distributions because different memory types are to be treated differently. Right now, we need complicated udev rules that e.g. check if we are running on s390x, on a physical system or on a virtualized system. But there is also