Re: [RFC V2 02/12] mm: Isolate HugeTLB allocations away from CDM nodes
On 02/01/2017 05:59 AM, Anshuman Khandual wrote: > So shall we write all these details in the comment section for each > patch after the SOB statement to be more visible ? Or some where > in-code documentation as FIXME or XXX or something. These are little > large paragraphs, hence was wondering. I would make an effort to convey a maximum amount of content in a minimal amount of words. :) But, yeah, it is pretty obvious that you've got too much in the cover letter and not enough in the patches descriptions. ... > * Page faults (which will probably use __GFP_THISNODE) cannot come from the > CDM nodes as they dont have any CPUs. Page faults happen on CPUs but they happen on VMAs that could be bound to a CDM node. We allocate based on the VMA policy first, the fall back to the default policy which is based on the CPU doing the fault if the VMA doesn't have a specific policy.
Re: [RFC V2 02/12] mm: Isolate HugeTLB allocations away from CDM nodes
On 01/31/2017 07:07 AM, Dave Hansen wrote: > On 01/30/2017 05:03 PM, Anshuman Khandual wrote: >> On 01/30/2017 10:49 PM, Dave Hansen wrote: >>> On 01/29/2017 07:35 PM, Anshuman Khandual wrote: HugeTLB allocation/release/accounting currently spans across all the nodes under N_MEMORY node mask. Coherent memory nodes should not be part of these allocations. So use system_ram() call to fetch system RAM only nodes on the platform which can then be used for HugeTLB allocation purpose instead of N_MEMORY node mask. This isolates coherent device memory nodes from HugeTLB allocations. >>> >>> Does this end up making it impossible to use hugetlbfs to access device >>> memory? >> >> Right, thats the implementation at the moment. But going forward if we need >> to have HugeTLB pages on the CDM node, then we can implement through the >> sysfs interface from individual NUMA node paths instead of changing the >> generic HugeTLB path. I wrote this up in the cover letter but should also >> have mentioned in the comment section of this patch as well. Does this >> approach look okay ? > > The cover letter is not the most approachable document I've ever seen. :) Hmm, So shall we write all these details in the comment section for each patch after the SOB statement to be more visible ? Or some where in-code documentation as FIXME or XXX or something. These are little large paragraphs, hence was wondering. > >> "Now, we ensure complete HugeTLB allocation isolation from CDM nodes. Going >> forward if we need to support HugeTLB allocation on CDM nodes on targeted >> basis, then we would have to enable those allocations through the >> /sys/devices/system/node/nodeN/hugepages/hugepages-16384kB/nr_hugepages >> interface while still ensuring isolation from other generic sysctl and >> /sys/kernel/mm/hugepages/hugepages-16384kB/nr_hugepages interfaces." > > That would be passable if that's the only way you can allocate hugetlbfs > pages. But we also have the fault-based allocations that can pull stuff > right out of the buddy allocator. This approach would break that path > entirely. There two distinct points which I think will prevent the problem you just mentioned. * No regular node has CDM memory in their fallback zone list. Hence any allocation attempt without __GFP_THISNODE will never go into CDM memory zones. If the allocation happens with __GFP_THISNODE flag it will only happen from the exact node. Remember we have removed CDM nodes from the global nodemask iterators. Then how can pre allocated reserve HugeTLB pages can come from CDM nodes ? * Page faults (which will probably use __GFP_THISNODE) cannot come from the CDM nodes as they dont have any CPUs. I did a quick scan of all the allocation paths leading upto the allocation functions alloc_pages_node() and __alloc_pages_node() inside the hugetlb.c file. Might be missing something here. > > FWIW, I think you really need to separate the true "CDM" stuff that's > *really* device-specific from the parts of this from which you really > just want to implement isolation. IIUC, are you suggesting something like a pure CDM HugeTLB implementation which is completely separated from the generic one ?
Re: [RFC V2 02/12] mm: Isolate HugeTLB allocations away from CDM nodes
On 01/30/2017 05:03 PM, Anshuman Khandual wrote: > On 01/30/2017 10:49 PM, Dave Hansen wrote: >> On 01/29/2017 07:35 PM, Anshuman Khandual wrote: >>> HugeTLB allocation/release/accounting currently spans across all the nodes >>> under N_MEMORY node mask. Coherent memory nodes should not be part of these >>> allocations. So use system_ram() call to fetch system RAM only nodes on the >>> platform which can then be used for HugeTLB allocation purpose instead of >>> N_MEMORY node mask. This isolates coherent device memory nodes from HugeTLB >>> allocations. >> >> Does this end up making it impossible to use hugetlbfs to access device >> memory? > > Right, thats the implementation at the moment. But going forward if we need > to have HugeTLB pages on the CDM node, then we can implement through the > sysfs interface from individual NUMA node paths instead of changing the > generic HugeTLB path. I wrote this up in the cover letter but should also > have mentioned in the comment section of this patch as well. Does this > approach look okay ? The cover letter is not the most approachable document I've ever seen. :) > "Now, we ensure complete HugeTLB allocation isolation from CDM nodes. Going > forward if we need to support HugeTLB allocation on CDM nodes on targeted > basis, then we would have to enable those allocations through the > /sys/devices/system/node/nodeN/hugepages/hugepages-16384kB/nr_hugepages > interface while still ensuring isolation from other generic sysctl and > /sys/kernel/mm/hugepages/hugepages-16384kB/nr_hugepages interfaces." That would be passable if that's the only way you can allocate hugetlbfs pages. But we also have the fault-based allocations that can pull stuff right out of the buddy allocator. This approach would break that path entirely. FWIW, I think you really need to separate the true "CDM" stuff that's *really* device-specific from the parts of this from which you really just want to implement isolation.
Re: [RFC V2 02/12] mm: Isolate HugeTLB allocations away from CDM nodes
On 01/30/2017 10:49 PM, Dave Hansen wrote: > On 01/29/2017 07:35 PM, Anshuman Khandual wrote: >> HugeTLB allocation/release/accounting currently spans across all the nodes >> under N_MEMORY node mask. Coherent memory nodes should not be part of these >> allocations. So use system_ram() call to fetch system RAM only nodes on the >> platform which can then be used for HugeTLB allocation purpose instead of >> N_MEMORY node mask. This isolates coherent device memory nodes from HugeTLB >> allocations. > > Does this end up making it impossible to use hugetlbfs to access device > memory? Right, thats the implementation at the moment. But going forward if we need to have HugeTLB pages on the CDM node, then we can implement through the sysfs interface from individual NUMA node paths instead of changing the generic HugeTLB path. I wrote this up in the cover letter but should also have mentioned in the comment section of this patch as well. Does this approach look okay ? "Now, we ensure complete HugeTLB allocation isolation from CDM nodes. Going forward if we need to support HugeTLB allocation on CDM nodes on targeted basis, then we would have to enable those allocations through the /sys/devices/system/node/nodeN/hugepages/hugepages-16384kB/nr_hugepages interface while still ensuring isolation from other generic sysctl and /sys/kernel/mm/hugepages/hugepages-16384kB/nr_hugepages interfaces."
Re: [RFC V2 02/12] mm: Isolate HugeTLB allocations away from CDM nodes
On 01/29/2017 07:35 PM, Anshuman Khandual wrote: > HugeTLB allocation/release/accounting currently spans across all the nodes > under N_MEMORY node mask. Coherent memory nodes should not be part of these > allocations. So use system_ram() call to fetch system RAM only nodes on the > platform which can then be used for HugeTLB allocation purpose instead of > N_MEMORY node mask. This isolates coherent device memory nodes from HugeTLB > allocations. Does this end up making it impossible to use hugetlbfs to access device memory?