Re: [PATCH v1 00/10] mm: online/offline 4MB chunks controlled by device driver

2018-07-18 Thread Michal Hocko
On Wed 18-07-18 15:47:33, David Hildenbrand wrote: [...] > So would you agree to change the comment in page-flags.h to something like > > "PG_reserved is set for special pages, that should never be touched > (read/written). Some of them might not even exist." Yes. Except you should mention that

Re: [PATCH v1 00/10] mm: online/offline 4MB chunks controlled by device driver

2018-07-18 Thread Michal Hocko
On Wed 18-07-18 15:47:33, David Hildenbrand wrote: [...] > So would you agree to change the comment in page-flags.h to something like > > "PG_reserved is set for special pages, that should never be touched > (read/written). Some of them might not even exist." Yes. Except you should mention that

Re: [PATCH v1 00/10] mm: online/offline 4MB chunks controlled by device driver

2018-07-18 Thread David Hildenbrand
On 18.07.2018 15:43, Michal Hocko wrote: > On Wed 18-07-18 15:39:29, David Hildenbrand wrote: >> On 18.07.2018 15:19, Michal Hocko wrote: >>> [got back to this really late. Sorry about that] >>> >>> On Thu 24-05-18 23:07:23, David Hildenbrand wrote: On 24.05.2018 16:22, Michal Hocko wrote:

Re: [PATCH v1 00/10] mm: online/offline 4MB chunks controlled by device driver

2018-07-18 Thread David Hildenbrand
On 18.07.2018 15:43, Michal Hocko wrote: > On Wed 18-07-18 15:39:29, David Hildenbrand wrote: >> On 18.07.2018 15:19, Michal Hocko wrote: >>> [got back to this really late. Sorry about that] >>> >>> On Thu 24-05-18 23:07:23, David Hildenbrand wrote: On 24.05.2018 16:22, Michal Hocko wrote:

Re: [PATCH v1 00/10] mm: online/offline 4MB chunks controlled by device driver

2018-07-18 Thread Michal Hocko
On Wed 18-07-18 15:39:29, David Hildenbrand wrote: > On 18.07.2018 15:19, Michal Hocko wrote: > > [got back to this really late. Sorry about that] > > > > On Thu 24-05-18 23:07:23, David Hildenbrand wrote: > >> On 24.05.2018 16:22, Michal Hocko wrote: > >>> I will go over the rest of the email

Re: [PATCH v1 00/10] mm: online/offline 4MB chunks controlled by device driver

2018-07-18 Thread Michal Hocko
On Wed 18-07-18 15:39:29, David Hildenbrand wrote: > On 18.07.2018 15:19, Michal Hocko wrote: > > [got back to this really late. Sorry about that] > > > > On Thu 24-05-18 23:07:23, David Hildenbrand wrote: > >> On 24.05.2018 16:22, Michal Hocko wrote: > >>> I will go over the rest of the email

Re: [PATCH v1 00/10] mm: online/offline 4MB chunks controlled by device driver

2018-07-18 Thread David Hildenbrand
On 18.07.2018 15:19, Michal Hocko wrote: > [got back to this really late. Sorry about that] > > On Thu 24-05-18 23:07:23, David Hildenbrand wrote: >> On 24.05.2018 16:22, Michal Hocko wrote: >>> I will go over the rest of the email later I just wanted to make this >>> point clear because I

Re: [PATCH v1 00/10] mm: online/offline 4MB chunks controlled by device driver

2018-07-18 Thread David Hildenbrand
On 18.07.2018 15:19, Michal Hocko wrote: > [got back to this really late. Sorry about that] > > On Thu 24-05-18 23:07:23, David Hildenbrand wrote: >> On 24.05.2018 16:22, Michal Hocko wrote: >>> I will go over the rest of the email later I just wanted to make this >>> point clear because I

Re: [PATCH v1 00/10] mm: online/offline 4MB chunks controlled by device driver

2018-07-18 Thread Michal Hocko
[got back to this really late. Sorry about that] On Thu 24-05-18 23:07:23, David Hildenbrand wrote: > On 24.05.2018 16:22, Michal Hocko wrote: > > I will go over the rest of the email later I just wanted to make this > > point clear because I suspect we are talking past each other. > > It sounds

Re: [PATCH v1 00/10] mm: online/offline 4MB chunks controlled by device driver

2018-07-18 Thread Michal Hocko
[got back to this really late. Sorry about that] On Thu 24-05-18 23:07:23, David Hildenbrand wrote: > On 24.05.2018 16:22, Michal Hocko wrote: > > I will go over the rest of the email later I just wanted to make this > > point clear because I suspect we are talking past each other. > > It sounds

Re: [PATCH v1 00/10] mm: online/offline 4MB chunks controlled by device driver

2018-07-18 Thread Michal Hocko
On Wed 18-07-18 11:56:29, David Hildenbrand wrote: > On 16.07.2018 22:05, Michal Hocko wrote: > > On Mon 16-07-18 21:48:59, David Hildenbrand wrote: > >> On 11.06.2018 14:33, David Hildenbrand wrote: > >>> On 11.06.2018 13:56, Michal Hocko wrote: > On Mon 11-06-18 13:53:49, David Hildenbrand

Re: [PATCH v1 00/10] mm: online/offline 4MB chunks controlled by device driver

2018-07-18 Thread Michal Hocko
On Wed 18-07-18 11:56:29, David Hildenbrand wrote: > On 16.07.2018 22:05, Michal Hocko wrote: > > On Mon 16-07-18 21:48:59, David Hildenbrand wrote: > >> On 11.06.2018 14:33, David Hildenbrand wrote: > >>> On 11.06.2018 13:56, Michal Hocko wrote: > On Mon 11-06-18 13:53:49, David Hildenbrand

Re: [PATCH v1 00/10] mm: online/offline 4MB chunks controlled by device driver

2018-07-18 Thread David Hildenbrand
On 16.07.2018 22:05, Michal Hocko wrote: > On Mon 16-07-18 21:48:59, David Hildenbrand wrote: >> On 11.06.2018 14:33, David Hildenbrand wrote: >>> On 11.06.2018 13:56, Michal Hocko wrote: On Mon 11-06-18 13:53:49, David Hildenbrand wrote: > On 24.05.2018 23:07, David Hildenbrand wrote:

Re: [PATCH v1 00/10] mm: online/offline 4MB chunks controlled by device driver

2018-07-18 Thread David Hildenbrand
On 16.07.2018 22:05, Michal Hocko wrote: > On Mon 16-07-18 21:48:59, David Hildenbrand wrote: >> On 11.06.2018 14:33, David Hildenbrand wrote: >>> On 11.06.2018 13:56, Michal Hocko wrote: On Mon 11-06-18 13:53:49, David Hildenbrand wrote: > On 24.05.2018 23:07, David Hildenbrand wrote:

Re: [PATCH v1 00/10] mm: online/offline 4MB chunks controlled by device driver

2018-07-16 Thread Michal Hocko
On Mon 16-07-18 21:48:59, David Hildenbrand wrote: > On 11.06.2018 14:33, David Hildenbrand wrote: > > On 11.06.2018 13:56, Michal Hocko wrote: > >> On Mon 11-06-18 13:53:49, David Hildenbrand wrote: > >>> On 24.05.2018 23:07, David Hildenbrand wrote: > On 24.05.2018 16:22, Michal Hocko

Re: [PATCH v1 00/10] mm: online/offline 4MB chunks controlled by device driver

2018-07-16 Thread Michal Hocko
On Mon 16-07-18 21:48:59, David Hildenbrand wrote: > On 11.06.2018 14:33, David Hildenbrand wrote: > > On 11.06.2018 13:56, Michal Hocko wrote: > >> On Mon 11-06-18 13:53:49, David Hildenbrand wrote: > >>> On 24.05.2018 23:07, David Hildenbrand wrote: > On 24.05.2018 16:22, Michal Hocko

Re: [PATCH v1 00/10] mm: online/offline 4MB chunks controlled by device driver

2018-07-16 Thread David Hildenbrand
On 11.06.2018 14:33, David Hildenbrand wrote: > On 11.06.2018 13:56, Michal Hocko wrote: >> On Mon 11-06-18 13:53:49, David Hildenbrand wrote: >>> On 24.05.2018 23:07, David Hildenbrand wrote: On 24.05.2018 16:22, Michal Hocko wrote: > I will go over the rest of the email later I just

Re: [PATCH v1 00/10] mm: online/offline 4MB chunks controlled by device driver

2018-07-16 Thread David Hildenbrand
On 11.06.2018 14:33, David Hildenbrand wrote: > On 11.06.2018 13:56, Michal Hocko wrote: >> On Mon 11-06-18 13:53:49, David Hildenbrand wrote: >>> On 24.05.2018 23:07, David Hildenbrand wrote: On 24.05.2018 16:22, Michal Hocko wrote: > I will go over the rest of the email later I just

Re: [PATCH v1 00/10] mm: online/offline 4MB chunks controlled by device driver

2018-06-11 Thread David Hildenbrand
On 11.06.2018 13:56, Michal Hocko wrote: > On Mon 11-06-18 13:53:49, David Hildenbrand wrote: >> On 24.05.2018 23:07, David Hildenbrand wrote: >>> On 24.05.2018 16:22, Michal Hocko wrote: I will go over the rest of the email later I just wanted to make this point clear because I suspect

Re: [PATCH v1 00/10] mm: online/offline 4MB chunks controlled by device driver

2018-06-11 Thread David Hildenbrand
On 11.06.2018 13:56, Michal Hocko wrote: > On Mon 11-06-18 13:53:49, David Hildenbrand wrote: >> On 24.05.2018 23:07, David Hildenbrand wrote: >>> On 24.05.2018 16:22, Michal Hocko wrote: I will go over the rest of the email later I just wanted to make this point clear because I suspect

Re: [PATCH v1 00/10] mm: online/offline 4MB chunks controlled by device driver

2018-06-11 Thread Michal Hocko
On Mon 11-06-18 13:53:49, David Hildenbrand wrote: > On 24.05.2018 23:07, David Hildenbrand wrote: > > On 24.05.2018 16:22, Michal Hocko wrote: > >> I will go over the rest of the email later I just wanted to make this > >> point clear because I suspect we are talking past each other. > > > > It

Re: [PATCH v1 00/10] mm: online/offline 4MB chunks controlled by device driver

2018-06-11 Thread Michal Hocko
On Mon 11-06-18 13:53:49, David Hildenbrand wrote: > On 24.05.2018 23:07, David Hildenbrand wrote: > > On 24.05.2018 16:22, Michal Hocko wrote: > >> I will go over the rest of the email later I just wanted to make this > >> point clear because I suspect we are talking past each other. > > > > It

Re: [PATCH v1 00/10] mm: online/offline 4MB chunks controlled by device driver

2018-06-11 Thread David Hildenbrand
On 24.05.2018 23:07, David Hildenbrand wrote: > On 24.05.2018 16:22, Michal Hocko wrote: >> I will go over the rest of the email later I just wanted to make this >> point clear because I suspect we are talking past each other. > > It sounds like we are now talking about how to solve the problem.

Re: [PATCH v1 00/10] mm: online/offline 4MB chunks controlled by device driver

2018-06-11 Thread David Hildenbrand
On 24.05.2018 23:07, David Hildenbrand wrote: > On 24.05.2018 16:22, Michal Hocko wrote: >> I will go over the rest of the email later I just wanted to make this >> point clear because I suspect we are talking past each other. > > It sounds like we are now talking about how to solve the problem.

Re: [PATCH v1 00/10] mm: online/offline 4MB chunks controlled by device driver

2018-05-28 Thread David Hildenbrand
On 28.05.2018 10:28, Dave Young wrote: > On 05/24/18 at 11:14am, David Hildenbrand wrote: >> On 24.05.2018 10:56, Dave Young wrote: >>> Hi, >>> >>> [snip] > >> For kdump and onlining/offlining code, we >> have to mark pages as offline before a new segment is visible to the >>

Re: [PATCH v1 00/10] mm: online/offline 4MB chunks controlled by device driver

2018-05-28 Thread David Hildenbrand
On 28.05.2018 10:28, Dave Young wrote: > On 05/24/18 at 11:14am, David Hildenbrand wrote: >> On 24.05.2018 10:56, Dave Young wrote: >>> Hi, >>> >>> [snip] > >> For kdump and onlining/offlining code, we >> have to mark pages as offline before a new segment is visible to the >>

Re: [PATCH v1 00/10] mm: online/offline 4MB chunks controlled by device driver

2018-05-28 Thread Dave Young
On 05/24/18 at 11:14am, David Hildenbrand wrote: > On 24.05.2018 10:56, Dave Young wrote: > > Hi, > > > > [snip] > >>> > For kdump and onlining/offlining code, we > have to mark pages as offline before a new segment is visible to the > system > (e.g. as these pages might not

Re: [PATCH v1 00/10] mm: online/offline 4MB chunks controlled by device driver

2018-05-28 Thread Dave Young
On 05/24/18 at 11:14am, David Hildenbrand wrote: > On 24.05.2018 10:56, Dave Young wrote: > > Hi, > > > > [snip] > >>> > For kdump and onlining/offlining code, we > have to mark pages as offline before a new segment is visible to the > system > (e.g. as these pages might not

Re: [PATCH v1 00/10] mm: online/offline 4MB chunks controlled by device driver

2018-05-25 Thread David Hildenbrand
>> So, no, virtio-mem is not a balloon driver :) > [...] 1. "hotplug should simply not depend on kdump at all" In theory yes. In the current state we already have to trigger kdump to reload whenever we add/remove a memory block. >>> >>> More details please. >> >> I just had

Re: [PATCH v1 00/10] mm: online/offline 4MB chunks controlled by device driver

2018-05-25 Thread David Hildenbrand
>> So, no, virtio-mem is not a balloon driver :) > [...] 1. "hotplug should simply not depend on kdump at all" In theory yes. In the current state we already have to trigger kdump to reload whenever we add/remove a memory block. >>> >>> More details please. >> >> I just had

Re: [PATCH v1 00/10] mm: online/offline 4MB chunks controlled by device driver

2018-05-24 Thread David Hildenbrand
On 24.05.2018 16:22, Michal Hocko wrote: > I will go over the rest of the email later I just wanted to make this > point clear because I suspect we are talking past each other. It sounds like we are now talking about how to solve the problem. I like that :) > > On Thu 24-05-18 16:04:38, David

Re: [PATCH v1 00/10] mm: online/offline 4MB chunks controlled by device driver

2018-05-24 Thread David Hildenbrand
On 24.05.2018 16:22, Michal Hocko wrote: > I will go over the rest of the email later I just wanted to make this > point clear because I suspect we are talking past each other. It sounds like we are now talking about how to solve the problem. I like that :) > > On Thu 24-05-18 16:04:38, David

Re: [PATCH v1 00/10] mm: online/offline 4MB chunks controlled by device driver

2018-05-24 Thread Michal Hocko
I will go over the rest of the email later I just wanted to make this point clear because I suspect we are talking past each other. On Thu 24-05-18 16:04:38, David Hildenbrand wrote: [...] > The point I was making is: I cannot allocate 8MB/128MB using the buddy > allocator. All I want to do is

Re: [PATCH v1 00/10] mm: online/offline 4MB chunks controlled by device driver

2018-05-24 Thread Michal Hocko
I will go over the rest of the email later I just wanted to make this point clear because I suspect we are talking past each other. On Thu 24-05-18 16:04:38, David Hildenbrand wrote: [...] > The point I was making is: I cannot allocate 8MB/128MB using the buddy > allocator. All I want to do is

Re: [PATCH v1 00/10] mm: online/offline 4MB chunks controlled by device driver

2018-05-24 Thread David Hildenbrand
>> You exactly describe what has been the case for way too long. But this >> is only the tip of the ice berg. Simply adding all memory to >> ZONE_MOVABLE is not going to work (we create an imbalance - e.g. page >> tables have to go into ZONE_NORMAL. this imbalance will have to be >> managed later

Re: [PATCH v1 00/10] mm: online/offline 4MB chunks controlled by device driver

2018-05-24 Thread David Hildenbrand
>> You exactly describe what has been the case for way too long. But this >> is only the tip of the ice berg. Simply adding all memory to >> ZONE_MOVABLE is not going to work (we create an imbalance - e.g. page >> tables have to go into ZONE_NORMAL. this imbalance will have to be >> managed later

Re: [PATCH v1 00/10] mm: online/offline 4MB chunks controlled by device driver

2018-05-24 Thread Michal Hocko
On Thu 24-05-18 12:45:50, David Hildenbrand wrote: > On 24.05.2018 11:31, Michal Hocko wrote: > > On Thu 24-05-18 10:31:30, David Hildenbrand wrote: [...] > >> Allowing to unplug such small chunks is actually the interesting thing. > > > > Not really. The vmemmap will stay behind and so you are

Re: [PATCH v1 00/10] mm: online/offline 4MB chunks controlled by device driver

2018-05-24 Thread Michal Hocko
On Thu 24-05-18 12:45:50, David Hildenbrand wrote: > On 24.05.2018 11:31, Michal Hocko wrote: > > On Thu 24-05-18 10:31:30, David Hildenbrand wrote: [...] > >> Allowing to unplug such small chunks is actually the interesting thing. > > > > Not really. The vmemmap will stay behind and so you are

Re: [PATCH v1 00/10] mm: online/offline 4MB chunks controlled by device driver

2018-05-24 Thread David Hildenbrand
On 24.05.2018 11:31, Michal Hocko wrote: > On Thu 24-05-18 10:31:30, David Hildenbrand wrote: >> On 24.05.2018 09:53, Michal Hocko wrote: >>> I've had some questions before and I am not sure they are fully covered. >>> At least not in the cover letter (I didn't get much further yet) which >>>

Re: [PATCH v1 00/10] mm: online/offline 4MB chunks controlled by device driver

2018-05-24 Thread David Hildenbrand
On 24.05.2018 11:31, Michal Hocko wrote: > On Thu 24-05-18 10:31:30, David Hildenbrand wrote: >> On 24.05.2018 09:53, Michal Hocko wrote: >>> I've had some questions before and I am not sure they are fully covered. >>> At least not in the cover letter (I didn't get much further yet) which >>>

Re: [PATCH v1 00/10] mm: online/offline 4MB chunks controlled by device driver

2018-05-24 Thread Michal Hocko
On Thu 24-05-18 10:31:30, David Hildenbrand wrote: > On 24.05.2018 09:53, Michal Hocko wrote: > > I've had some questions before and I am not sure they are fully covered. > > At least not in the cover letter (I didn't get much further yet) which > > should give us a highlevel overview of the

Re: [PATCH v1 00/10] mm: online/offline 4MB chunks controlled by device driver

2018-05-24 Thread Michal Hocko
On Thu 24-05-18 10:31:30, David Hildenbrand wrote: > On 24.05.2018 09:53, Michal Hocko wrote: > > I've had some questions before and I am not sure they are fully covered. > > At least not in the cover letter (I didn't get much further yet) which > > should give us a highlevel overview of the

Re: [PATCH v1 00/10] mm: online/offline 4MB chunks controlled by device driver

2018-05-24 Thread David Hildenbrand
On 24.05.2018 10:56, Dave Young wrote: > Hi, > > [snip] >>> For kdump and onlining/offlining code, we have to mark pages as offline before a new segment is visible to the system (e.g. as these pages might not be backed by real memory in the hypervisor). >>> >>> Please expand on the

Re: [PATCH v1 00/10] mm: online/offline 4MB chunks controlled by device driver

2018-05-24 Thread David Hildenbrand
On 24.05.2018 10:56, Dave Young wrote: > Hi, > > [snip] >>> For kdump and onlining/offlining code, we have to mark pages as offline before a new segment is visible to the system (e.g. as these pages might not be backed by real memory in the hypervisor). >>> >>> Please expand on the

Re: [PATCH v1 00/10] mm: online/offline 4MB chunks controlled by device driver

2018-05-24 Thread Dave Young
Hi, [snip] > > > >> For kdump and onlining/offlining code, we > >> have to mark pages as offline before a new segment is visible to the system > >> (e.g. as these pages might not be backed by real memory in the hypervisor). > > > > Please expand on the kdump part. That is really confusing

Re: [PATCH v1 00/10] mm: online/offline 4MB chunks controlled by device driver

2018-05-24 Thread Dave Young
Hi, [snip] > > > >> For kdump and onlining/offlining code, we > >> have to mark pages as offline before a new segment is visible to the system > >> (e.g. as these pages might not be backed by real memory in the hypervisor). > > > > Please expand on the kdump part. That is really confusing

Re: [PATCH v1 00/10] mm: online/offline 4MB chunks controlled by device driver

2018-05-24 Thread David Hildenbrand
On 24.05.2018 09:53, Michal Hocko wrote: > I've had some questions before and I am not sure they are fully covered. > At least not in the cover letter (I didn't get much further yet) which > should give us a highlevel overview of the feature. Sure, I can give you more details. Adding all details

Re: [PATCH v1 00/10] mm: online/offline 4MB chunks controlled by device driver

2018-05-24 Thread David Hildenbrand
On 24.05.2018 09:53, Michal Hocko wrote: > I've had some questions before and I am not sure they are fully covered. > At least not in the cover letter (I didn't get much further yet) which > should give us a highlevel overview of the feature. Sure, I can give you more details. Adding all details

Re: [PATCH v1 00/10] mm: online/offline 4MB chunks controlled by device driver

2018-05-24 Thread Michal Hocko
I've had some questions before and I am not sure they are fully covered. At least not in the cover letter (I didn't get much further yet) which should give us a highlevel overview of the feature. On Wed 23-05-18 17:11:41, David Hildenbrand wrote: > This is now the !RFC version. I did some

Re: [PATCH v1 00/10] mm: online/offline 4MB chunks controlled by device driver

2018-05-24 Thread Michal Hocko
I've had some questions before and I am not sure they are fully covered. At least not in the cover letter (I didn't get much further yet) which should give us a highlevel overview of the feature. On Wed 23-05-18 17:11:41, David Hildenbrand wrote: > This is now the !RFC version. I did some

[PATCH v1 00/10] mm: online/offline 4MB chunks controlled by device driver

2018-05-23 Thread David Hildenbrand
This is now the !RFC version. I did some additional tests and inspected all memory notifiers. At least page_ext and kasan need fixes. == I am right now working on a paravirtualized memory device ("virtio-mem"). These devices control a memory region and the amount of memory available via

[PATCH v1 00/10] mm: online/offline 4MB chunks controlled by device driver

2018-05-23 Thread David Hildenbrand
This is now the !RFC version. I did some additional tests and inspected all memory notifiers. At least page_ext and kasan need fixes. == I am right now working on a paravirtualized memory device ("virtio-mem"). These devices control a memory region and the amount of memory available via