Re: [PATCH 00/11] x86, memblock: Allocate memory near kernel image before SRAT parsed.
Hello, On Mon, Sep 09, 2013 at 07:56:34PM +0800, Wanpeng Li wrote: > If allocate from low to high as what this patchset done will occupy the > precious memory you mentioned? Yeah, and that'd be the reason why this behavior is dependent on a kernel option. That said, allocating some megs on top of kernel isn't a big deal. The wretched ISA DMA is mostly gone now and some megs isn't gonna hurt 32bit DMAs in any noticeable way. I wouldn't be too surprised if nobody notices after switching the default behavior to allocate early mem close to kernel. Maybe the only case which might be impacted is 32bit highmem configs, but they're messed up no matter what anyway and even they shouldn't be affected noticeably if large mapping is in use. Thanks. -- tejun -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH 00/11] x86, memblock: Allocate memory near kernel image before SRAT parsed.
Hello, On Mon, Sep 09, 2013 at 07:56:34PM +0800, Wanpeng Li wrote: If allocate from low to high as what this patchset done will occupy the precious memory you mentioned? Yeah, and that'd be the reason why this behavior is dependent on a kernel option. That said, allocating some megs on top of kernel isn't a big deal. The wretched ISA DMA is mostly gone now and some megs isn't gonna hurt 32bit DMAs in any noticeable way. I wouldn't be too surprised if nobody notices after switching the default behavior to allocate early mem close to kernel. Maybe the only case which might be impacted is 32bit highmem configs, but they're messed up no matter what anyway and even they shouldn't be affected noticeably if large mapping is in use. Thanks. -- tejun -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH 00/11] x86, memblock: Allocate memory near kernel image before SRAT parsed.
Specifically there are a bunch of things which need to be below a certain address (which one varies.) Tejun Heo wrote: >Hello, Wanpeng. > >On Fri, Sep 06, 2013 at 04:58:11PM +0800, Wanpeng Li wrote: >> What's the root reason memblock alloc from high to low? To reduce >> fragmentation or ... > >Because low memory tends to be more precious, it's just easier to pack >everything towards the top so that we don't have to worry about which >zone to use for allocation and fallback logic. > >Thanks. -- Sent from my mobile phone. Please pardon brevity and lack of formatting. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH 00/11] x86, memblock: Allocate memory near kernel image before SRAT parsed.
Hello, Wanpeng. On Fri, Sep 06, 2013 at 04:58:11PM +0800, Wanpeng Li wrote: > What's the root reason memblock alloc from high to low? To reduce > fragmentation or ... Because low memory tends to be more precious, it's just easier to pack everything towards the top so that we don't have to worry about which zone to use for allocation and fallback logic. Thanks. -- tejun -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH 00/11] x86, memblock: Allocate memory near kernel image before SRAT parsed.
Hello, Wanpeng. On Fri, Sep 06, 2013 at 04:58:11PM +0800, Wanpeng Li wrote: What's the root reason memblock alloc from high to low? To reduce fragmentation or ... Because low memory tends to be more precious, it's just easier to pack everything towards the top so that we don't have to worry about which zone to use for allocation and fallback logic. Thanks. -- tejun -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH 00/11] x86, memblock: Allocate memory near kernel image before SRAT parsed.
Specifically there are a bunch of things which need to be below a certain address (which one varies.) Tejun Heo t...@kernel.org wrote: Hello, Wanpeng. On Fri, Sep 06, 2013 at 04:58:11PM +0800, Wanpeng Li wrote: What's the root reason memblock alloc from high to low? To reduce fragmentation or ... Because low memory tends to be more precious, it's just easier to pack everything towards the top so that we don't have to worry about which zone to use for allocation and fallback logic. Thanks. -- Sent from my mobile phone. Please pardon brevity and lack of formatting. -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH 00/11] x86, memblock: Allocate memory near kernel image before SRAT parsed.
Hi tj, On 09/05/2013 03:22 AM, Tejun Heo wrote: .. I'm expectedly happier with this approach but some overall review points. * I think patch splitting went a bit too far. e.g. it doesn't make much sense or helps anything to split "introduction of a param" from "the param doing something". * I think it's a lot more complex than necessary. Just implement a single function - memblock_alloc_bottom_up(@start) where specifying MEMBLOCK_ALLOC_ANYWHERE restores top down behavior and do memblock_alloc_bottom_up(end_of_kernel) early during boot. If the bottom up mode is set, just try allocating bottom up from the specified address and if that fails do normal top down allocation. No need to meddle with the callers. The only change necessary (well, aside from the reordering) outside memblock is adding two calls to the above function. * I don't think "order" is the right word here. "direction" probably fits a lot better. Thanks for the advices. I'll try to simply the code and send a new patch-set soon. Thanks. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH 00/11] x86, memblock: Allocate memory near kernel image before SRAT parsed.
Hi tj, On 09/05/2013 03:22 AM, Tejun Heo wrote: .. I'm expectedly happier with this approach but some overall review points. * I think patch splitting went a bit too far. e.g. it doesn't make much sense or helps anything to split introduction of a param from the param doing something. * I think it's a lot more complex than necessary. Just implement a single function - memblock_alloc_bottom_up(@start) where specifying MEMBLOCK_ALLOC_ANYWHERE restores top down behavior and do memblock_alloc_bottom_up(end_of_kernel) early during boot. If the bottom up mode is set, just try allocating bottom up from the specified address and if that fails do normal top down allocation. No need to meddle with the callers. The only change necessary (well, aside from the reordering) outside memblock is adding two calls to the above function. * I don't think order is the right word here. direction probably fits a lot better. Thanks for the advices. I'll try to simply the code and send a new patch-set soon. Thanks. -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH 00/11] x86, memblock: Allocate memory near kernel image before SRAT parsed.
Hello, On Tue, Aug 27, 2013 at 05:37:37PM +0800, Tang Chen wrote: > 1. Make memblock be able to allocate memory from low address to high address. >Also introduce low limit to prevent memblock allocating memory too low. > > 2. Improve init_mem_mapping() to support allocate page tables from low > address >to high address. > > 3. Introduce "movablenode" boot option to enable and disable this > functionality. > > PS: Reordering of relocate_initrd() and reserve_crashkernel() has not been > done > yet. acpi_initrd_override() needs to access initrd with virtual address. > So > relocate_initrd() must be done before acpi_initrd_override(). I'm expectedly happier with this approach but some overall review points. * I think patch splitting went a bit too far. e.g. it doesn't make much sense or helps anything to split "introduction of a param" from "the param doing something". * I think it's a lot more complex than necessary. Just implement a single function - memblock_alloc_bottom_up(@start) where specifying MEMBLOCK_ALLOC_ANYWHERE restores top down behavior and do memblock_alloc_bottom_up(end_of_kernel) early during boot. If the bottom up mode is set, just try allocating bottom up from the specified address and if that fails do normal top down allocation. No need to meddle with the callers. The only change necessary (well, aside from the reordering) outside memblock is adding two calls to the above function. * I don't think "order" is the right word here. "direction" probably fits a lot better. Thanks. -- tejun -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH 00/11] x86, memblock: Allocate memory near kernel image before SRAT parsed.
Hello, On Tue, Aug 27, 2013 at 05:37:37PM +0800, Tang Chen wrote: 1. Make memblock be able to allocate memory from low address to high address. Also introduce low limit to prevent memblock allocating memory too low. 2. Improve init_mem_mapping() to support allocate page tables from low address to high address. 3. Introduce movablenode boot option to enable and disable this functionality. PS: Reordering of relocate_initrd() and reserve_crashkernel() has not been done yet. acpi_initrd_override() needs to access initrd with virtual address. So relocate_initrd() must be done before acpi_initrd_override(). I'm expectedly happier with this approach but some overall review points. * I think patch splitting went a bit too far. e.g. it doesn't make much sense or helps anything to split introduction of a param from the param doing something. * I think it's a lot more complex than necessary. Just implement a single function - memblock_alloc_bottom_up(@start) where specifying MEMBLOCK_ALLOC_ANYWHERE restores top down behavior and do memblock_alloc_bottom_up(end_of_kernel) early during boot. If the bottom up mode is set, just try allocating bottom up from the specified address and if that fails do normal top down allocation. No need to meddle with the callers. The only change necessary (well, aside from the reordering) outside memblock is adding two calls to the above function. * I don't think order is the right word here. direction probably fits a lot better. Thanks. -- tejun -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH 00/11] x86, memblock: Allocate memory near kernel image before SRAT parsed.
Hi guys, Any comment to this patch-set ? And shall agree on using this solution suggested by Tejun ? Thanks. On 08/27/2013 05:37 PM, Tang Chen wrote: This patch-set is based on tj's suggestion, and not fully tested. Just for review and discussion. [Problem] The current Linux cannot migrate pages used by the kerenl because of the kernel direct mapping. In Linux kernel space, va = pa + PAGE_OFFSET. When the pa is changed, we cannot simply update the pagetable and keep the va unmodified. So the kernel pages are not migratable. There are also some other issues will cause the kernel pages not migratable. For example, the physical address may be cached somewhere and will be used. It is not to update all the caches. When doing memory hotplug in Linux, we first migrate all the pages in one memory device somewhere else, and then remove the device. But if pages are used by the kernel, they are not migratable. As a result, memory used by the kernel cannot be hot-removed. Modifying the kernel direct mapping mechanism is too difficult to do. And it may cause the kernel performance down and unstable. So we use the following way to do memory hotplug. [What we are doing] In Linux, memory in one numa node is divided into several zones. One of the zones is ZONE_MOVABLE, which the kernel won't use. In order to implement memory hotplug in Linux, we are going to arrange all hotpluggable memory in ZONE_MOVABLE so that the kernel won't use these memory. To do this, we need ACPI's help. In ACPI, SRAT(System Resource Affinity Table) contains NUMA info. The memory affinities in SRAT record every memory range in the system, and also, flags specifying if the memory range is hotpluggable. (Please refer to ACPI spec 5.0 5.2.16) With the help of SRAT, we have to do the following two things to achieve our goal: 1. When doing memory hot-add, allow the users arranging hotpluggable as ZONE_MOVABLE. (This has been done by the MOVABLE_NODE functionality in Linux.) 2. when the system is booting, prevent bootmem allocator from allocating hotpluggable memory for the kernel before the memory initialization finishes. The problem 2 is the key problem we are going to solve. But before solving it, we need some preparation. Please see below. [Preparation] Bootloader has to load the kernel image into memory. And this memory must be unhotpluggable. We cannot prevent this anyway. So in a memory hotplug system, we can assume any node the kernel resides in is not hotpluggable. Before SRAT is parsed, we don't know which memory ranges are hotpluggable. But memblock has already started to work. In the current kernel, memblock allocates the following memory before SRAT is parsed: setup_arch() |->memblock_x86_fill()/* memblock is ready */ |.. |->early_reserve_e820_mpc_new() /* allocate memory under 1MB */ |->reserve_real_mode()/* allocate memory under 1MB */ |->init_mem_mapping() /* allocate page tables, about 2MB to map 1GB memory */ |->dma_contiguous_reserve() /* specified by user, should be low */ |->setup_log_buf()/* specified by user, several mega bytes */ |->relocate_initrd() /* could be large, but will be freed after boot, should reorder */ |->acpi_initrd_override() /* several mega bytes */ |->reserve_crashkernel() /* could be large, should reorder */ |.. |->initmem_init() /* Parse SRAT */ According to Tejun's advice, before SRAT is parsed, we should try our best to allocate memory near the kernel image. Since the whole node the kernel resides in won't be hotpluggable, and for a modern server, a node may have at least 16GB memory, allocating several mega bytes memory around the kernel image won't cross to hotpluggable memory. [About this patch-set] So this patch-set does the following: 1. Make memblock be able to allocate memory from low address to high address. Also introduce low limit to prevent memblock allocating memory too low. 2. Improve init_mem_mapping() to support allocate page tables from low address to high address. 3. Introduce "movablenode" boot option to enable and disable this functionality. PS: Reordering of relocate_initrd() and reserve_crashkernel() has not been done yet. acpi_initrd_override() needs to access initrd with virtual address. So relocate_initrd() must be done before acpi_initrd_override(). Tang Chen (11): memblock: Rename current_limit to current_limit_high in memblock. memblock: Rename memblock_set_current_limit() to memblock_set_current_limit_high(). memblock: Introduce lowest limit in memblock. memblock: Introduce memblock_set_current_limit_low() to set lower limit of memblock. memblock: Introduce allocation order to memblock. memblock: Improve memblock to support allocation from lower address. x86, memblock: Set lowest limit for memblock_alloc_base_nid(). x86, acpi, memblock: Use
Re: [PATCH 00/11] x86, memblock: Allocate memory near kernel image before SRAT parsed.
Hi guys, Any comment to this patch-set ? And shall agree on using this solution suggested by Tejun ? Thanks. On 08/27/2013 05:37 PM, Tang Chen wrote: This patch-set is based on tj's suggestion, and not fully tested. Just for review and discussion. [Problem] The current Linux cannot migrate pages used by the kerenl because of the kernel direct mapping. In Linux kernel space, va = pa + PAGE_OFFSET. When the pa is changed, we cannot simply update the pagetable and keep the va unmodified. So the kernel pages are not migratable. There are also some other issues will cause the kernel pages not migratable. For example, the physical address may be cached somewhere and will be used. It is not to update all the caches. When doing memory hotplug in Linux, we first migrate all the pages in one memory device somewhere else, and then remove the device. But if pages are used by the kernel, they are not migratable. As a result, memory used by the kernel cannot be hot-removed. Modifying the kernel direct mapping mechanism is too difficult to do. And it may cause the kernel performance down and unstable. So we use the following way to do memory hotplug. [What we are doing] In Linux, memory in one numa node is divided into several zones. One of the zones is ZONE_MOVABLE, which the kernel won't use. In order to implement memory hotplug in Linux, we are going to arrange all hotpluggable memory in ZONE_MOVABLE so that the kernel won't use these memory. To do this, we need ACPI's help. In ACPI, SRAT(System Resource Affinity Table) contains NUMA info. The memory affinities in SRAT record every memory range in the system, and also, flags specifying if the memory range is hotpluggable. (Please refer to ACPI spec 5.0 5.2.16) With the help of SRAT, we have to do the following two things to achieve our goal: 1. When doing memory hot-add, allow the users arranging hotpluggable as ZONE_MOVABLE. (This has been done by the MOVABLE_NODE functionality in Linux.) 2. when the system is booting, prevent bootmem allocator from allocating hotpluggable memory for the kernel before the memory initialization finishes. The problem 2 is the key problem we are going to solve. But before solving it, we need some preparation. Please see below. [Preparation] Bootloader has to load the kernel image into memory. And this memory must be unhotpluggable. We cannot prevent this anyway. So in a memory hotplug system, we can assume any node the kernel resides in is not hotpluggable. Before SRAT is parsed, we don't know which memory ranges are hotpluggable. But memblock has already started to work. In the current kernel, memblock allocates the following memory before SRAT is parsed: setup_arch() |-memblock_x86_fill()/* memblock is ready */ |.. |-early_reserve_e820_mpc_new() /* allocate memory under 1MB */ |-reserve_real_mode()/* allocate memory under 1MB */ |-init_mem_mapping() /* allocate page tables, about 2MB to map 1GB memory */ |-dma_contiguous_reserve() /* specified by user, should be low */ |-setup_log_buf()/* specified by user, several mega bytes */ |-relocate_initrd() /* could be large, but will be freed after boot, should reorder */ |-acpi_initrd_override() /* several mega bytes */ |-reserve_crashkernel() /* could be large, should reorder */ |.. |-initmem_init() /* Parse SRAT */ According to Tejun's advice, before SRAT is parsed, we should try our best to allocate memory near the kernel image. Since the whole node the kernel resides in won't be hotpluggable, and for a modern server, a node may have at least 16GB memory, allocating several mega bytes memory around the kernel image won't cross to hotpluggable memory. [About this patch-set] So this patch-set does the following: 1. Make memblock be able to allocate memory from low address to high address. Also introduce low limit to prevent memblock allocating memory too low. 2. Improve init_mem_mapping() to support allocate page tables from low address to high address. 3. Introduce movablenode boot option to enable and disable this functionality. PS: Reordering of relocate_initrd() and reserve_crashkernel() has not been done yet. acpi_initrd_override() needs to access initrd with virtual address. So relocate_initrd() must be done before acpi_initrd_override(). Tang Chen (11): memblock: Rename current_limit to current_limit_high in memblock. memblock: Rename memblock_set_current_limit() to memblock_set_current_limit_high(). memblock: Introduce lowest limit in memblock. memblock: Introduce memblock_set_current_limit_low() to set lower limit of memblock. memblock: Introduce allocation order to memblock. memblock: Improve memblock to support allocation from lower address. x86, memblock: Set lowest limit for memblock_alloc_base_nid(). x86, acpi, memblock: Use
Re: [PATCH 00/11] x86, memblock: Allocate memory near kernel image before SRAT parsed.
Hi Wanpeng, On 08/29/2013 09:36 AM, Wanpeng Li wrote: .. Hi tj, Sorry for the trouble. Please refer to the following branch: https://github.com/imtangchen/linux.git movablenode-boot-option Could you post your testcase? So I can test it on x86 and powerpc machines. Sure. Some simple testcases: 1. Boot the kernel without movablenode boot option, check if the memory mapping is initialized as before, high to low. 2. Boot the kernel with movablenode boot option, check if the memory mapping is initialized as before, low to high. 3. With movablenode, check if the memory allocation is from high to low after SRAT is parsed. 4. Check if we can do acpi_initrd_override normally with and without movablenode. And the memory allocation is from low to high, near the end of kernel image. 5. with movablenode, check if crashkernel boot option works normally. (This may consume a lot of memory, but should work normally.) 6. With movablenode, check if relocate_initrd() works normally. (This may consume a lot of memory, but should work normally.) 7. With movablenode, check if kexec could locate the kernel to higher memory. (This may consume hotplug memory if higher memory is hotpluggable, but should work normally.) Please do the above tests with and without the following config options: 1. CONFIG_MOVABLE_NODE 2. CONFIG_ACPI_INITRD_OVERRIDE Thanks for the testing. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH 00/11] x86, memblock: Allocate memory near kernel image before SRAT parsed.
On 08/28/2013 11:19 PM, Tejun Heo wrote: .. Doesn't apply to -master, -next or tip. Again, can you please include which tree and git commit the patches are against in the patch description? How is one supposed to know on top of which tree you're working? It is in your benefit to make things easier for the prosepct reviewers. Trying to guess and apply the patches to different devel branches and failing isn't productive and frustates your prospect reviewers who would of course have negative pre-perception going into the review and this isn't the first time this issue was raised either. Hi tj, Sorry for the trouble. Please refer to the following branch: https://github.com/imtangchen/linux.git movablenode-boot-option Thanks. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH 00/11] x86, memblock: Allocate memory near kernel image before SRAT parsed.
On Tue, Aug 27, 2013 at 05:37:37PM +0800, Tang Chen wrote:
> Tang Chen (11):
> memblock: Rename current_limit to current_limit_high in memblock.
> memblock: Rename memblock_set_current_limit() to
> memblock_set_current_limit_high().
> memblock: Introduce lowest limit in memblock.
> memblock: Introduce memblock_set_current_limit_low() to set lower
> limit of memblock.
> memblock: Introduce allocation order to memblock.
> memblock: Improve memblock to support allocation from lower address.
> x86, memblock: Set lowest limit for memblock_alloc_base_nid().
> x86, acpi, memblock: Use __memblock_alloc_base() in
> acpi_initrd_override()
> mem-hotplug: Introduce movablenode boot option to {en|dis}able using
> SRAT.
> x86, mem-hotplug: Support initialize page tables from low to high.
> x86, mem_hotplug: Allocate memory near kernel image before SRAT is
> parsed.
Doesn't apply to -master, -next or tip. Again, can you please include
which tree and git commit the patches are against in the patch
description? How is one supposed to know on top of which tree you're
working? It is in your benefit to make things easier for the prosepct
reviewers. Trying to guess and apply the patches to different devel
branches and failing isn't productive and frustates your prospect
reviewers who would of course have negative pre-perception going into
the review and this isn't the first time this issue was raised either.
--
tejun
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH 00/11] x86, memblock: Allocate memory near kernel image before SRAT parsed.
Hi Wanpeng On 08/28/2013 04:03 PM, Wanpeng Li wrote: Hi Tang, .. [About this patch-set] So this patch-set does the following: 1. Make memblock be able to allocate memory from low address to high address. I want to know if there is fragmentation degree difference here? Before this patch-set, we mapped memory like this: 1. [0, ISA_END_ADDRESS), 2. [ISA_END_ADDRESS, round_down(max_addr, PMD_SIZE)), from top downwards, 3. [round_down(max_addr, PMD_SIZE), max_addr) After this patch-set, when movablenode is enabled, it is like: 1. [round_up(_end, PMD_SIZE), max_addr), from _end upwards, 2. [ISA_END_ADDRESS, round_up(_end, PMD_SIZE)), 3. [0, ISA_END_ADDRESS) All the boundaries are aligned with PMD_SIZE. I think it is the same. Thanks. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH 00/11] x86, memblock: Allocate memory near kernel image before SRAT parsed.
Hi Wanpeng On 08/28/2013 04:03 PM, Wanpeng Li wrote: Hi Tang, .. [About this patch-set] So this patch-set does the following: 1. Make memblock be able to allocate memory from low address to high address. I want to know if there is fragmentation degree difference here? Before this patch-set, we mapped memory like this: 1. [0, ISA_END_ADDRESS), 2. [ISA_END_ADDRESS, round_down(max_addr, PMD_SIZE)), from top downwards, 3. [round_down(max_addr, PMD_SIZE), max_addr) After this patch-set, when movablenode is enabled, it is like: 1. [round_up(_end, PMD_SIZE), max_addr), from _end upwards, 2. [ISA_END_ADDRESS, round_up(_end, PMD_SIZE)), 3. [0, ISA_END_ADDRESS) All the boundaries are aligned with PMD_SIZE. I think it is the same. Thanks. -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH 00/11] x86, memblock: Allocate memory near kernel image before SRAT parsed.
On Tue, Aug 27, 2013 at 05:37:37PM +0800, Tang Chen wrote:
Tang Chen (11):
memblock: Rename current_limit to current_limit_high in memblock.
memblock: Rename memblock_set_current_limit() to
memblock_set_current_limit_high().
memblock: Introduce lowest limit in memblock.
memblock: Introduce memblock_set_current_limit_low() to set lower
limit of memblock.
memblock: Introduce allocation order to memblock.
memblock: Improve memblock to support allocation from lower address.
x86, memblock: Set lowest limit for memblock_alloc_base_nid().
x86, acpi, memblock: Use __memblock_alloc_base() in
acpi_initrd_override()
mem-hotplug: Introduce movablenode boot option to {en|dis}able using
SRAT.
x86, mem-hotplug: Support initialize page tables from low to high.
x86, mem_hotplug: Allocate memory near kernel image before SRAT is
parsed.
Doesn't apply to -master, -next or tip. Again, can you please include
which tree and git commit the patches are against in the patch
description? How is one supposed to know on top of which tree you're
working? It is in your benefit to make things easier for the prosepct
reviewers. Trying to guess and apply the patches to different devel
branches and failing isn't productive and frustates your prospect
reviewers who would of course have negative pre-perception going into
the review and this isn't the first time this issue was raised either.
--
tejun
--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH 00/11] x86, memblock: Allocate memory near kernel image before SRAT parsed.
On 08/28/2013 11:19 PM, Tejun Heo wrote: .. Doesn't apply to -master, -next or tip. Again, can you please include which tree and git commit the patches are against in the patch description? How is one supposed to know on top of which tree you're working? It is in your benefit to make things easier for the prosepct reviewers. Trying to guess and apply the patches to different devel branches and failing isn't productive and frustates your prospect reviewers who would of course have negative pre-perception going into the review and this isn't the first time this issue was raised either. Hi tj, Sorry for the trouble. Please refer to the following branch: https://github.com/imtangchen/linux.git movablenode-boot-option Thanks. -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH 00/11] x86, memblock: Allocate memory near kernel image before SRAT parsed.
Hi Wanpeng, On 08/29/2013 09:36 AM, Wanpeng Li wrote: .. Hi tj, Sorry for the trouble. Please refer to the following branch: https://github.com/imtangchen/linux.git movablenode-boot-option Could you post your testcase? So I can test it on x86 and powerpc machines. Sure. Some simple testcases: 1. Boot the kernel without movablenode boot option, check if the memory mapping is initialized as before, high to low. 2. Boot the kernel with movablenode boot option, check if the memory mapping is initialized as before, low to high. 3. With movablenode, check if the memory allocation is from high to low after SRAT is parsed. 4. Check if we can do acpi_initrd_override normally with and without movablenode. And the memory allocation is from low to high, near the end of kernel image. 5. with movablenode, check if crashkernel boot option works normally. (This may consume a lot of memory, but should work normally.) 6. With movablenode, check if relocate_initrd() works normally. (This may consume a lot of memory, but should work normally.) 7. With movablenode, check if kexec could locate the kernel to higher memory. (This may consume hotplug memory if higher memory is hotpluggable, but should work normally.) Please do the above tests with and without the following config options: 1. CONFIG_MOVABLE_NODE 2. CONFIG_ACPI_INITRD_OVERRIDE Thanks for the testing. -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
[PATCH 00/11] x86, memblock: Allocate memory near kernel image before SRAT parsed.
This patch-set is based on tj's suggestion, and not fully tested.
Just for review and discussion.
[Problem]
The current Linux cannot migrate pages used by the kerenl because
of the kernel direct mapping. In Linux kernel space, va = pa + PAGE_OFFSET.
When the pa is changed, we cannot simply update the pagetable and
keep the va unmodified. So the kernel pages are not migratable.
There are also some other issues will cause the kernel pages not migratable.
For example, the physical address may be cached somewhere and will be used.
It is not to update all the caches.
When doing memory hotplug in Linux, we first migrate all the pages in one
memory device somewhere else, and then remove the device. But if pages are
used by the kernel, they are not migratable. As a result, memory used by
the kernel cannot be hot-removed.
Modifying the kernel direct mapping mechanism is too difficult to do. And
it may cause the kernel performance down and unstable. So we use the following
way to do memory hotplug.
[What we are doing]
In Linux, memory in one numa node is divided into several zones. One of the
zones is ZONE_MOVABLE, which the kernel won't use.
In order to implement memory hotplug in Linux, we are going to arrange all
hotpluggable memory in ZONE_MOVABLE so that the kernel won't use these memory.
To do this, we need ACPI's help.
In ACPI, SRAT(System Resource Affinity Table) contains NUMA info. The memory
affinities in SRAT record every memory range in the system, and also, flags
specifying if the memory range is hotpluggable.
(Please refer to ACPI spec 5.0 5.2.16)
With the help of SRAT, we have to do the following two things to achieve our
goal:
1. When doing memory hot-add, allow the users arranging hotpluggable as
ZONE_MOVABLE.
(This has been done by the MOVABLE_NODE functionality in Linux.)
2. when the system is booting, prevent bootmem allocator from allocating
hotpluggable memory for the kernel before the memory initialization
finishes.
The problem 2 is the key problem we are going to solve. But before solving it,
we need some preparation. Please see below.
[Preparation]
Bootloader has to load the kernel image into memory. And this memory must be
unhotpluggable. We cannot prevent this anyway. So in a memory hotplug system,
we can assume any node the kernel resides in is not hotpluggable.
Before SRAT is parsed, we don't know which memory ranges are hotpluggable. But
memblock has already started to work. In the current kernel, memblock allocates
the following memory before SRAT is parsed:
setup_arch()
|->memblock_x86_fill()/* memblock is ready */
|..
|->early_reserve_e820_mpc_new() /* allocate memory under 1MB */
|->reserve_real_mode()/* allocate memory under 1MB */
|->init_mem_mapping() /* allocate page tables, about 2MB to map
1GB memory */
|->dma_contiguous_reserve() /* specified by user, should be low */
|->setup_log_buf()/* specified by user, several mega bytes */
|->relocate_initrd() /* could be large, but will be freed after
boot, should reorder */
|->acpi_initrd_override() /* several mega bytes */
|->reserve_crashkernel() /* could be large, should reorder */
|..
|->initmem_init() /* Parse SRAT */
According to Tejun's advice, before SRAT is parsed, we should try our best to
allocate memory near the kernel image. Since the whole node the kernel resides
in won't be hotpluggable, and for a modern server, a node may have at least 16GB
memory, allocating several mega bytes memory around the kernel image won't cross
to hotpluggable memory.
[About this patch-set]
So this patch-set does the following:
1. Make memblock be able to allocate memory from low address to high address.
Also introduce low limit to prevent memblock allocating memory too low.
2. Improve init_mem_mapping() to support allocate page tables from low address
to high address.
3. Introduce "movablenode" boot option to enable and disable this functionality.
PS: Reordering of relocate_initrd() and reserve_crashkernel() has not been done
yet. acpi_initrd_override() needs to access initrd with virtual address. So
relocate_initrd() must be done before acpi_initrd_override().
Tang Chen (11):
memblock: Rename current_limit to current_limit_high in memblock.
memblock: Rename memblock_set_current_limit() to
memblock_set_current_limit_high().
memblock: Introduce lowest limit in memblock.
memblock: Introduce memblock_set_current_limit_low() to set lower
limit of memblock.
memblock: Introduce allocation order to memblock.
memblock: Improve memblock to support allocation from lower address.
x86, memblock: Set lowest limit for memblock_alloc_base_nid().
x86, acpi, memblock: Use __memblock_alloc_base() in
acpi_initrd_override()
mem-hotplug: Introduce movablenode boot option to {en|dis}able using
SRAT.
x86, mem-hotplug: Support initialize
[PATCH 00/11] x86, memblock: Allocate memory near kernel image before SRAT parsed.
This patch-set is based on tj's suggestion, and not fully tested.
Just for review and discussion.
[Problem]
The current Linux cannot migrate pages used by the kerenl because
of the kernel direct mapping. In Linux kernel space, va = pa + PAGE_OFFSET.
When the pa is changed, we cannot simply update the pagetable and
keep the va unmodified. So the kernel pages are not migratable.
There are also some other issues will cause the kernel pages not migratable.
For example, the physical address may be cached somewhere and will be used.
It is not to update all the caches.
When doing memory hotplug in Linux, we first migrate all the pages in one
memory device somewhere else, and then remove the device. But if pages are
used by the kernel, they are not migratable. As a result, memory used by
the kernel cannot be hot-removed.
Modifying the kernel direct mapping mechanism is too difficult to do. And
it may cause the kernel performance down and unstable. So we use the following
way to do memory hotplug.
[What we are doing]
In Linux, memory in one numa node is divided into several zones. One of the
zones is ZONE_MOVABLE, which the kernel won't use.
In order to implement memory hotplug in Linux, we are going to arrange all
hotpluggable memory in ZONE_MOVABLE so that the kernel won't use these memory.
To do this, we need ACPI's help.
In ACPI, SRAT(System Resource Affinity Table) contains NUMA info. The memory
affinities in SRAT record every memory range in the system, and also, flags
specifying if the memory range is hotpluggable.
(Please refer to ACPI spec 5.0 5.2.16)
With the help of SRAT, we have to do the following two things to achieve our
goal:
1. When doing memory hot-add, allow the users arranging hotpluggable as
ZONE_MOVABLE.
(This has been done by the MOVABLE_NODE functionality in Linux.)
2. when the system is booting, prevent bootmem allocator from allocating
hotpluggable memory for the kernel before the memory initialization
finishes.
The problem 2 is the key problem we are going to solve. But before solving it,
we need some preparation. Please see below.
[Preparation]
Bootloader has to load the kernel image into memory. And this memory must be
unhotpluggable. We cannot prevent this anyway. So in a memory hotplug system,
we can assume any node the kernel resides in is not hotpluggable.
Before SRAT is parsed, we don't know which memory ranges are hotpluggable. But
memblock has already started to work. In the current kernel, memblock allocates
the following memory before SRAT is parsed:
setup_arch()
|-memblock_x86_fill()/* memblock is ready */
|..
|-early_reserve_e820_mpc_new() /* allocate memory under 1MB */
|-reserve_real_mode()/* allocate memory under 1MB */
|-init_mem_mapping() /* allocate page tables, about 2MB to map
1GB memory */
|-dma_contiguous_reserve() /* specified by user, should be low */
|-setup_log_buf()/* specified by user, several mega bytes */
|-relocate_initrd() /* could be large, but will be freed after
boot, should reorder */
|-acpi_initrd_override() /* several mega bytes */
|-reserve_crashkernel() /* could be large, should reorder */
|..
|-initmem_init() /* Parse SRAT */
According to Tejun's advice, before SRAT is parsed, we should try our best to
allocate memory near the kernel image. Since the whole node the kernel resides
in won't be hotpluggable, and for a modern server, a node may have at least 16GB
memory, allocating several mega bytes memory around the kernel image won't cross
to hotpluggable memory.
[About this patch-set]
So this patch-set does the following:
1. Make memblock be able to allocate memory from low address to high address.
Also introduce low limit to prevent memblock allocating memory too low.
2. Improve init_mem_mapping() to support allocate page tables from low address
to high address.
3. Introduce movablenode boot option to enable and disable this functionality.
PS: Reordering of relocate_initrd() and reserve_crashkernel() has not been done
yet. acpi_initrd_override() needs to access initrd with virtual address. So
relocate_initrd() must be done before acpi_initrd_override().
Tang Chen (11):
memblock: Rename current_limit to current_limit_high in memblock.
memblock: Rename memblock_set_current_limit() to
memblock_set_current_limit_high().
memblock: Introduce lowest limit in memblock.
memblock: Introduce memblock_set_current_limit_low() to set lower
limit of memblock.
memblock: Introduce allocation order to memblock.
memblock: Improve memblock to support allocation from lower address.
x86, memblock: Set lowest limit for memblock_alloc_base_nid().
x86, acpi, memblock: Use __memblock_alloc_base() in
acpi_initrd_override()
mem-hotplug: Introduce movablenode boot option to {en|dis}able using
SRAT.
x86, mem-hotplug: Support initialize page tables
