Re: [PATCH v3 0/6] Introduce ZONE_CMA

[Joonsoo Kim]

On Mon, Jun 27, 2016 at 09:25:45PM +1000, Balbir Singh wrote:
> 
> 
> On 26/05/16 16:22, js1...@gmail.com wrote:
> > From: Joonsoo Kim 
> > 
> > Hello,
> > 
> > Changes from v2
> > o Rebase on next-20160525
> > o No other changes except following description
> > 
> > There was a discussion with Mel [1] after LSF/MM 2016. I could summarise
> > it to help merge decision but it's better to read by yourself since
> > if I summarise it, it would be biased for me. But, if anyone hope
> > the summary, I will do it. :)
> > 
> > Anyway, Mel's position on this patchset seems to be neutral. He said:
> > "I'm not going to outright NAK your series but I won't ACK it either"
> > 
> > We can fix the problems with any approach but I hope to go a new zone
> > approach because it is less error-prone. It reduces some corner case
> > handling for now and remove need for potential corner case handling to fix
> > problems.
> > 
> > Note that our company is already using ZONE_CMA for a years and
> > there is no problem.
> > 
> > If anyone has a different opinion, please let me know and let's discuss
> > together.
> > 
> > Andrew, if there is something to do for merge, please let me know.
> > 
> > [1] https://lkml.kernel.org/r/20160425053653.GA25662@js1304-P5Q-DELUXE
> > 
> > Changes from v1
> > o Separate some patches which deserve to submit independently
> > o Modify description to reflect current kernel state
> > (e.g. high-order watermark problem disappeared by Mel's work)
> > o Don't increase SECTION_SIZE_BITS to make a room in page flags
> > (detailed reason is on the patch that adds ZONE_CMA)
> > o Adjust ZONE_CMA population code
> > 
> > This series try to solve problems of current CMA implementation.
> > 
> > CMA is introduced to provide physically contiguous pages at runtime
> > without exclusive reserved memory area. But, current implementation
> > works like as previous reserved memory approach, because freepages
> > on CMA region are used only if there is no movable freepage. In other
> > words, freepages on CMA region are only used as fallback. In that
> > situation where freepages on CMA region are used as fallback, kswapd
> > would be woken up easily since there is no unmovable and reclaimable
> > freepage, too. If kswapd starts to reclaim memory, fallback allocation
> > to MIGRATE_CMA doesn't occur any more since movable freepages are
> > already refilled by kswapd and then most of freepage on CMA are left
> > to be in free. This situation looks like exclusive reserved memory case.
> 
> I am afraid I don't understand the problem statement completely understand.
> Is this the ALLOC_CMA case or the !ALLOC_CMA one? I also think one other

It's caused by the mixed usage of these flags, not caused by one
specific flags.

> problem is that in my experience and observation all CMA allocations seem
> to come from one node-- the highest node on the system
> 
> > 
> > In my experiment, I found that if system memory has 1024 MB memory and
> > 512 MB is reserved for CMA, kswapd is mostly woken up when roughly 512 MB
> > free memory is left. Detailed reason is that for keeping enough free
> > memory for unmovable and reclaimable allocation, kswapd uses below
> > equation when calculating free memory and it easily go under the watermark.
> > 
> > Free memory for unmovable and reclaimable = Free total - Free CMA pages
> > 
> > This is derivated from the property of CMA freepage that CMA freepage
> > can't be used for unmovable and reclaimable allocation.
> > 
> > Anyway, in this case, kswapd are woken up when (FreeTotal - FreeCMA)
> > is lower than low watermark and tries to make free memory until
> > (FreeTotal - FreeCMA) is higher than high watermark. That results
> > in that FreeTotal is moving around 512MB boundary consistently. It
> > then means that we can't utilize full memory capacity.
> > 
> 
> OK.. so you are suggesting that we are under-utilizing the memory in the
> CMA region?

That's right.

> > To fix this problem, I submitted some patches [1] about 10 months ago,
> > but, found some more problems to be fixed before solving this problem.
> > It requires many hooks in allocator hotpath so some developers doesn't
> > like it. Instead, some of them suggest different approach [2] to fix
> > all the problems related to CMA, that is, introducing a new zone to deal
> > with free CMA pages. I agree that it is the best way to go so implement
> > here. Although properties of ZONE_MOVABLE and ZONE_CMA is similar, I
> > decide to add a new zone rather than piggyback on ZONE_MOVABLE since
> > they have some differences. First, reserved CMA pages should not be
> > offlined.
> 
> Why? Why are they special? Even if they are offlined by user action,
> one would expect the following to occur
> 
> 1. User would mark/release the cma region associated with them
> 2. User would then hotplug the memory

CMA region is reserved at booting time and used until system
shutdown. Hotplug CMA region isn't possible, yet. 

Re: [PATCH v3 0/6] Introduce ZONE_CMA

[Joonsoo Kim]

On Mon, Jun 27, 2016 at 09:25:45PM +1000, Balbir Singh wrote:
> 
> 
> On 26/05/16 16:22, js1...@gmail.com wrote:
> > From: Joonsoo Kim 
> > 
> > Hello,
> > 
> > Changes from v2
> > o Rebase on next-20160525
> > o No other changes except following description
> > 
> > There was a discussion with Mel [1] after LSF/MM 2016. I could summarise
> > it to help merge decision but it's better to read by yourself since
> > if I summarise it, it would be biased for me. But, if anyone hope
> > the summary, I will do it. :)
> > 
> > Anyway, Mel's position on this patchset seems to be neutral. He said:
> > "I'm not going to outright NAK your series but I won't ACK it either"
> > 
> > We can fix the problems with any approach but I hope to go a new zone
> > approach because it is less error-prone. It reduces some corner case
> > handling for now and remove need for potential corner case handling to fix
> > problems.
> > 
> > Note that our company is already using ZONE_CMA for a years and
> > there is no problem.
> > 
> > If anyone has a different opinion, please let me know and let's discuss
> > together.
> > 
> > Andrew, if there is something to do for merge, please let me know.
> > 
> > [1] https://lkml.kernel.org/r/20160425053653.GA25662@js1304-P5Q-DELUXE
> > 
> > Changes from v1
> > o Separate some patches which deserve to submit independently
> > o Modify description to reflect current kernel state
> > (e.g. high-order watermark problem disappeared by Mel's work)
> > o Don't increase SECTION_SIZE_BITS to make a room in page flags
> > (detailed reason is on the patch that adds ZONE_CMA)
> > o Adjust ZONE_CMA population code
> > 
> > This series try to solve problems of current CMA implementation.
> > 
> > CMA is introduced to provide physically contiguous pages at runtime
> > without exclusive reserved memory area. But, current implementation
> > works like as previous reserved memory approach, because freepages
> > on CMA region are used only if there is no movable freepage. In other
> > words, freepages on CMA region are only used as fallback. In that
> > situation where freepages on CMA region are used as fallback, kswapd
> > would be woken up easily since there is no unmovable and reclaimable
> > freepage, too. If kswapd starts to reclaim memory, fallback allocation
> > to MIGRATE_CMA doesn't occur any more since movable freepages are
> > already refilled by kswapd and then most of freepage on CMA are left
> > to be in free. This situation looks like exclusive reserved memory case.
> 
> I am afraid I don't understand the problem statement completely understand.
> Is this the ALLOC_CMA case or the !ALLOC_CMA one? I also think one other

It's caused by the mixed usage of these flags, not caused by one
specific flags.

> problem is that in my experience and observation all CMA allocations seem
> to come from one node-- the highest node on the system
> 
> > 
> > In my experiment, I found that if system memory has 1024 MB memory and
> > 512 MB is reserved for CMA, kswapd is mostly woken up when roughly 512 MB
> > free memory is left. Detailed reason is that for keeping enough free
> > memory for unmovable and reclaimable allocation, kswapd uses below
> > equation when calculating free memory and it easily go under the watermark.
> > 
> > Free memory for unmovable and reclaimable = Free total - Free CMA pages
> > 
> > This is derivated from the property of CMA freepage that CMA freepage
> > can't be used for unmovable and reclaimable allocation.
> > 
> > Anyway, in this case, kswapd are woken up when (FreeTotal - FreeCMA)
> > is lower than low watermark and tries to make free memory until
> > (FreeTotal - FreeCMA) is higher than high watermark. That results
> > in that FreeTotal is moving around 512MB boundary consistently. It
> > then means that we can't utilize full memory capacity.
> > 
> 
> OK.. so you are suggesting that we are under-utilizing the memory in the
> CMA region?

That's right.

> > To fix this problem, I submitted some patches [1] about 10 months ago,
> > but, found some more problems to be fixed before solving this problem.
> > It requires many hooks in allocator hotpath so some developers doesn't
> > like it. Instead, some of them suggest different approach [2] to fix
> > all the problems related to CMA, that is, introducing a new zone to deal
> > with free CMA pages. I agree that it is the best way to go so implement
> > here. Although properties of ZONE_MOVABLE and ZONE_CMA is similar, I
> > decide to add a new zone rather than piggyback on ZONE_MOVABLE since
> > they have some differences. First, reserved CMA pages should not be
> > offlined.
> 
> Why? Why are they special? Even if they are offlined by user action,
> one would expect the following to occur
> 
> 1. User would mark/release the cma region associated with them
> 2. User would then hotplug the memory

CMA region is reserved at booting time and used until system
shutdown. Hotplug CMA region isn't possible, yet. Later, we would
handle 

Re: [PATCH v3 0/6] Introduce ZONE_CMA

[Balbir Singh]

On 26/05/16 16:22, js1...@gmail.com wrote:
> From: Joonsoo Kim 
> 
> Hello,
> 
> Changes from v2
> o Rebase on next-20160525
> o No other changes except following description
> 
> There was a discussion with Mel [1] after LSF/MM 2016. I could summarise
> it to help merge decision but it's better to read by yourself since
> if I summarise it, it would be biased for me. But, if anyone hope
> the summary, I will do it. :)
> 
> Anyway, Mel's position on this patchset seems to be neutral. He said:
> "I'm not going to outright NAK your series but I won't ACK it either"
> 
> We can fix the problems with any approach but I hope to go a new zone
> approach because it is less error-prone. It reduces some corner case
> handling for now and remove need for potential corner case handling to fix
> problems.
> 
> Note that our company is already using ZONE_CMA for a years and
> there is no problem.
> 
> If anyone has a different opinion, please let me know and let's discuss
> together.
> 
> Andrew, if there is something to do for merge, please let me know.
> 
> [1] https://lkml.kernel.org/r/20160425053653.GA25662@js1304-P5Q-DELUXE
> 
> Changes from v1
> o Separate some patches which deserve to submit independently
> o Modify description to reflect current kernel state
> (e.g. high-order watermark problem disappeared by Mel's work)
> o Don't increase SECTION_SIZE_BITS to make a room in page flags
> (detailed reason is on the patch that adds ZONE_CMA)
> o Adjust ZONE_CMA population code
> 
> This series try to solve problems of current CMA implementation.
> 
> CMA is introduced to provide physically contiguous pages at runtime
> without exclusive reserved memory area. But, current implementation
> works like as previous reserved memory approach, because freepages
> on CMA region are used only if there is no movable freepage. In other
> words, freepages on CMA region are only used as fallback. In that
> situation where freepages on CMA region are used as fallback, kswapd
> would be woken up easily since there is no unmovable and reclaimable
> freepage, too. If kswapd starts to reclaim memory, fallback allocation
> to MIGRATE_CMA doesn't occur any more since movable freepages are
> already refilled by kswapd and then most of freepage on CMA are left
> to be in free. This situation looks like exclusive reserved memory case.

I am afraid I don't understand the problem statement completely understand.
Is this the ALLOC_CMA case or the !ALLOC_CMA one? I also think one other
problem is that in my experience and observation all CMA allocations seem
to come from one node-- the highest node on the system

> 
> In my experiment, I found that if system memory has 1024 MB memory and
> 512 MB is reserved for CMA, kswapd is mostly woken up when roughly 512 MB
> free memory is left. Detailed reason is that for keeping enough free
> memory for unmovable and reclaimable allocation, kswapd uses below
> equation when calculating free memory and it easily go under the watermark.
> 
> Free memory for unmovable and reclaimable = Free total - Free CMA pages
> 
> This is derivated from the property of CMA freepage that CMA freepage
> can't be used for unmovable and reclaimable allocation.
> 
> Anyway, in this case, kswapd are woken up when (FreeTotal - FreeCMA)
> is lower than low watermark and tries to make free memory until
> (FreeTotal - FreeCMA) is higher than high watermark. That results
> in that FreeTotal is moving around 512MB boundary consistently. It
> then means that we can't utilize full memory capacity.
> 

OK.. so you are suggesting that we are under-utilizing the memory in the
CMA region?

> To fix this problem, I submitted some patches [1] about 10 months ago,
> but, found some more problems to be fixed before solving this problem.
> It requires many hooks in allocator hotpath so some developers doesn't
> like it. Instead, some of them suggest different approach [2] to fix
> all the problems related to CMA, that is, introducing a new zone to deal
> with free CMA pages. I agree that it is the best way to go so implement
> here. Although properties of ZONE_MOVABLE and ZONE_CMA is similar, I
> decide to add a new zone rather than piggyback on ZONE_MOVABLE since
> they have some differences. First, reserved CMA pages should not be
> offlined.

Why? Why are they special? Even if they are offlined by user action,
one would expect the following to occur

1. User would mark/release the cma region associated with them
2. User would then hotplug the memory

> If freepage for CMA is managed by ZONE_MOVABLE, we need to keep
> MIGRATE_CMA migratetype and insert many hooks on memory hotplug code
> to distiguish hotpluggable memory and reserved memory for CMA in the same
> zone. It would make memory hotplug code which is already complicated
> more complicated.

Again why treat it special, one could potentially deny the hotplug based
on the knowledge of where the CMA region is allocated from

> Second, cma_alloc() can be 

Re: [PATCH v3 0/6] Introduce ZONE_CMA

[Balbir Singh]

On 26/05/16 16:22, js1...@gmail.com wrote:
> From: Joonsoo Kim 
> 
> Hello,
> 
> Changes from v2
> o Rebase on next-20160525
> o No other changes except following description
> 
> There was a discussion with Mel [1] after LSF/MM 2016. I could summarise
> it to help merge decision but it's better to read by yourself since
> if I summarise it, it would be biased for me. But, if anyone hope
> the summary, I will do it. :)
> 
> Anyway, Mel's position on this patchset seems to be neutral. He said:
> "I'm not going to outright NAK your series but I won't ACK it either"
> 
> We can fix the problems with any approach but I hope to go a new zone
> approach because it is less error-prone. It reduces some corner case
> handling for now and remove need for potential corner case handling to fix
> problems.
> 
> Note that our company is already using ZONE_CMA for a years and
> there is no problem.
> 
> If anyone has a different opinion, please let me know and let's discuss
> together.
> 
> Andrew, if there is something to do for merge, please let me know.
> 
> [1] https://lkml.kernel.org/r/20160425053653.GA25662@js1304-P5Q-DELUXE
> 
> Changes from v1
> o Separate some patches which deserve to submit independently
> o Modify description to reflect current kernel state
> (e.g. high-order watermark problem disappeared by Mel's work)
> o Don't increase SECTION_SIZE_BITS to make a room in page flags
> (detailed reason is on the patch that adds ZONE_CMA)
> o Adjust ZONE_CMA population code
> 
> This series try to solve problems of current CMA implementation.
> 
> CMA is introduced to provide physically contiguous pages at runtime
> without exclusive reserved memory area. But, current implementation
> works like as previous reserved memory approach, because freepages
> on CMA region are used only if there is no movable freepage. In other
> words, freepages on CMA region are only used as fallback. In that
> situation where freepages on CMA region are used as fallback, kswapd
> would be woken up easily since there is no unmovable and reclaimable
> freepage, too. If kswapd starts to reclaim memory, fallback allocation
> to MIGRATE_CMA doesn't occur any more since movable freepages are
> already refilled by kswapd and then most of freepage on CMA are left
> to be in free. This situation looks like exclusive reserved memory case.

I am afraid I don't understand the problem statement completely understand.
Is this the ALLOC_CMA case or the !ALLOC_CMA one? I also think one other
problem is that in my experience and observation all CMA allocations seem
to come from one node-- the highest node on the system

> 
> In my experiment, I found that if system memory has 1024 MB memory and
> 512 MB is reserved for CMA, kswapd is mostly woken up when roughly 512 MB
> free memory is left. Detailed reason is that for keeping enough free
> memory for unmovable and reclaimable allocation, kswapd uses below
> equation when calculating free memory and it easily go under the watermark.
> 
> Free memory for unmovable and reclaimable = Free total - Free CMA pages
> 
> This is derivated from the property of CMA freepage that CMA freepage
> can't be used for unmovable and reclaimable allocation.
> 
> Anyway, in this case, kswapd are woken up when (FreeTotal - FreeCMA)
> is lower than low watermark and tries to make free memory until
> (FreeTotal - FreeCMA) is higher than high watermark. That results
> in that FreeTotal is moving around 512MB boundary consistently. It
> then means that we can't utilize full memory capacity.
> 

OK.. so you are suggesting that we are under-utilizing the memory in the
CMA region?

> To fix this problem, I submitted some patches [1] about 10 months ago,
> but, found some more problems to be fixed before solving this problem.
> It requires many hooks in allocator hotpath so some developers doesn't
> like it. Instead, some of them suggest different approach [2] to fix
> all the problems related to CMA, that is, introducing a new zone to deal
> with free CMA pages. I agree that it is the best way to go so implement
> here. Although properties of ZONE_MOVABLE and ZONE_CMA is similar, I
> decide to add a new zone rather than piggyback on ZONE_MOVABLE since
> they have some differences. First, reserved CMA pages should not be
> offlined.

Why? Why are they special? Even if they are offlined by user action,
one would expect the following to occur

1. User would mark/release the cma region associated with them
2. User would then hotplug the memory

> If freepage for CMA is managed by ZONE_MOVABLE, we need to keep
> MIGRATE_CMA migratetype and insert many hooks on memory hotplug code
> to distiguish hotpluggable memory and reserved memory for CMA in the same
> zone. It would make memory hotplug code which is already complicated
> more complicated.

Again why treat it special, one could potentially deny the hotplug based
on the knowledge of where the CMA region is allocated from

> Second, cma_alloc() can be called more frequently

Re: [PATCH v3 0/6] Introduce ZONE_CMA

[Joonsoo Kim]

On Tue, Jun 21, 2016 at 10:08:24AM +0800, Chen Feng wrote:
> 
> 
> On 2016/6/20 14:48, Joonsoo Kim wrote:
> > On Fri, Jun 17, 2016 at 03:38:49PM +0800, Chen Feng wrote:
> >> Hi Kim & feng,
> >>
> >> Thanks for the share. In our platform also has the same use case.
> >>
> >> We only let the alloc with GFP_HIGHUSER_MOVABLE in memory.c to use cma 
> >> memory.
> >>
> >> If we add zone_cma, It seems can resolve the cma migrate issue.
> >>
> >> But when free_hot_cold_page, we need let the cma page goto system directly 
> >> not the pcp.
> >> It can be fail while cma_alloc and cma_release. If we alloc the whole cma 
> >> pages which
> >> declared before.
> > 
> > Hmm...I'm not sure I understand your explanation. So, if I miss
> > something, please let me know. We calls drain_all_pages() when
> > isolating pageblock and alloc_contig_range() also has one
> > drain_all_pages() calls to drain pcp pages. And, after pageblock isolation,
> > freed pages belonging to MIGRATE_ISOLATE pageblock will go to the
> > buddy directly so there would be no problem you mentioned. Isn't it?
> > 
> Yes, you are right.
> 
> I mean if the we free cma page to pcp-list, it will goto the migrate_movable 
> list.
> 
> Then the alloc with movable flag can use the cma memory from the list with 
> buffered_rmqueue.
> 
> But that's not what we want. It will cause the migrate fail if all movable 
> alloc can use cma memory.

Yes, if you modify current kernel code to allow cma pages only for
GFP_HIGHUSER_MOVABLE in memory.c, there are some corner cases and some of cma
pages would be allocated for !GFP_HIGHUSER_MOVABLE. One possible site is
pcp list as you mentioned and the other site is on compaction.

If we uses ZONE_CMA, there is no such problem, because freepages on
pcp list on ZONE_CMA are allocated only when GFP_HIGHUSER_MOVABLE requset
comes.

Thanks.

Re: [PATCH v3 0/6] Introduce ZONE_CMA

[Joonsoo Kim]

On Tue, Jun 21, 2016 at 10:08:24AM +0800, Chen Feng wrote:
> 
> 
> On 2016/6/20 14:48, Joonsoo Kim wrote:
> > On Fri, Jun 17, 2016 at 03:38:49PM +0800, Chen Feng wrote:
> >> Hi Kim & feng,
> >>
> >> Thanks for the share. In our platform also has the same use case.
> >>
> >> We only let the alloc with GFP_HIGHUSER_MOVABLE in memory.c to use cma 
> >> memory.
> >>
> >> If we add zone_cma, It seems can resolve the cma migrate issue.
> >>
> >> But when free_hot_cold_page, we need let the cma page goto system directly 
> >> not the pcp.
> >> It can be fail while cma_alloc and cma_release. If we alloc the whole cma 
> >> pages which
> >> declared before.
> > 
> > Hmm...I'm not sure I understand your explanation. So, if I miss
> > something, please let me know. We calls drain_all_pages() when
> > isolating pageblock and alloc_contig_range() also has one
> > drain_all_pages() calls to drain pcp pages. And, after pageblock isolation,
> > freed pages belonging to MIGRATE_ISOLATE pageblock will go to the
> > buddy directly so there would be no problem you mentioned. Isn't it?
> > 
> Yes, you are right.
> 
> I mean if the we free cma page to pcp-list, it will goto the migrate_movable 
> list.
> 
> Then the alloc with movable flag can use the cma memory from the list with 
> buffered_rmqueue.
> 
> But that's not what we want. It will cause the migrate fail if all movable 
> alloc can use cma memory.

Yes, if you modify current kernel code to allow cma pages only for
GFP_HIGHUSER_MOVABLE in memory.c, there are some corner cases and some of cma
pages would be allocated for !GFP_HIGHUSER_MOVABLE. One possible site is
pcp list as you mentioned and the other site is on compaction.

If we uses ZONE_CMA, there is no such problem, because freepages on
pcp list on ZONE_CMA are allocated only when GFP_HIGHUSER_MOVABLE requset
comes.

Thanks.

Re: [PATCH v3 0/6] Introduce ZONE_CMA

[Chen Feng]

On 2016/6/20 14:48, Joonsoo Kim wrote:
> On Fri, Jun 17, 2016 at 03:38:49PM +0800, Chen Feng wrote:
>> Hi Kim & feng,
>>
>> Thanks for the share. In our platform also has the same use case.
>>
>> We only let the alloc with GFP_HIGHUSER_MOVABLE in memory.c to use cma 
>> memory.
>>
>> If we add zone_cma, It seems can resolve the cma migrate issue.
>>
>> But when free_hot_cold_page, we need let the cma page goto system directly 
>> not the pcp.
>> It can be fail while cma_alloc and cma_release. If we alloc the whole cma 
>> pages which
>> declared before.
> 
> Hmm...I'm not sure I understand your explanation. So, if I miss
> something, please let me know. We calls drain_all_pages() when
> isolating pageblock and alloc_contig_range() also has one
> drain_all_pages() calls to drain pcp pages. And, after pageblock isolation,
> freed pages belonging to MIGRATE_ISOLATE pageblock will go to the
> buddy directly so there would be no problem you mentioned. Isn't it?
> 
Yes, you are right.

I mean if the we free cma page to pcp-list, it will goto the migrate_movable 
list.

Then the alloc with movable flag can use the cma memory from the list with 
buffered_rmqueue.

But that's not what we want. It will cause the migrate fail if all movable 
alloc can use cma memory.

If I am wrong, please correct me.

Thanks.

> Thanks.
> 
> .
> 

Re: [PATCH v3 0/6] Introduce ZONE_CMA

[Chen Feng]

On 2016/6/20 14:48, Joonsoo Kim wrote:
> On Fri, Jun 17, 2016 at 03:38:49PM +0800, Chen Feng wrote:
>> Hi Kim & feng,
>>
>> Thanks for the share. In our platform also has the same use case.
>>
>> We only let the alloc with GFP_HIGHUSER_MOVABLE in memory.c to use cma 
>> memory.
>>
>> If we add zone_cma, It seems can resolve the cma migrate issue.
>>
>> But when free_hot_cold_page, we need let the cma page goto system directly 
>> not the pcp.
>> It can be fail while cma_alloc and cma_release. If we alloc the whole cma 
>> pages which
>> declared before.
> 
> Hmm...I'm not sure I understand your explanation. So, if I miss
> something, please let me know. We calls drain_all_pages() when
> isolating pageblock and alloc_contig_range() also has one
> drain_all_pages() calls to drain pcp pages. And, after pageblock isolation,
> freed pages belonging to MIGRATE_ISOLATE pageblock will go to the
> buddy directly so there would be no problem you mentioned. Isn't it?
> 
Yes, you are right.

I mean if the we free cma page to pcp-list, it will goto the migrate_movable 
list.

Then the alloc with movable flag can use the cma memory from the list with 
buffered_rmqueue.

But that's not what we want. It will cause the migrate fail if all movable 
alloc can use cma memory.

If I am wrong, please correct me.

Thanks.

> Thanks.
> 
> .
> 

Re: [PATCH v3 0/6] Introduce ZONE_CMA

[Joonsoo Kim]

On Fri, Jun 17, 2016 at 03:38:49PM +0800, Chen Feng wrote:
> Hi Kim & feng,
> 
> Thanks for the share. In our platform also has the same use case.
> 
> We only let the alloc with GFP_HIGHUSER_MOVABLE in memory.c to use cma memory.
> 
> If we add zone_cma, It seems can resolve the cma migrate issue.
> 
> But when free_hot_cold_page, we need let the cma page goto system directly 
> not the pcp.
> It can be fail while cma_alloc and cma_release. If we alloc the whole cma 
> pages which
> declared before.

Hmm...I'm not sure I understand your explanation. So, if I miss
something, please let me know. We calls drain_all_pages() when
isolating pageblock and alloc_contig_range() also has one
drain_all_pages() calls to drain pcp pages. And, after pageblock isolation,
freed pages belonging to MIGRATE_ISOLATE pageblock will go to the
buddy directly so there would be no problem you mentioned. Isn't it?

Thanks.

Re: [PATCH v3 0/6] Introduce ZONE_CMA

[Joonsoo Kim]

On Fri, Jun 17, 2016 at 03:38:49PM +0800, Chen Feng wrote:
> Hi Kim & feng,
> 
> Thanks for the share. In our platform also has the same use case.
> 
> We only let the alloc with GFP_HIGHUSER_MOVABLE in memory.c to use cma memory.
> 
> If we add zone_cma, It seems can resolve the cma migrate issue.
> 
> But when free_hot_cold_page, we need let the cma page goto system directly 
> not the pcp.
> It can be fail while cma_alloc and cma_release. If we alloc the whole cma 
> pages which
> declared before.

Hmm...I'm not sure I understand your explanation. So, if I miss
something, please let me know. We calls drain_all_pages() when
isolating pageblock and alloc_contig_range() also has one
drain_all_pages() calls to drain pcp pages. And, after pageblock isolation,
freed pages belonging to MIGRATE_ISOLATE pageblock will go to the
buddy directly so there would be no problem you mentioned. Isn't it?

Thanks.

Re: [PATCH v3 0/6] Introduce ZONE_CMA

[Chen Feng]

Hi Kim & feng,

Thanks for the share. In our platform also has the same use case.

We only let the alloc with GFP_HIGHUSER_MOVABLE in memory.c to use cma memory.

If we add zone_cma, It seems can resolve the cma migrate issue.

But when free_hot_cold_page, we need let the cma page goto system directly not 
the pcp.
It can be fail while cma_alloc and cma_release. If we alloc the whole cma pages 
which
declared before.

On 2016/5/27 15:27, Feng Tang wrote:
> On Fri, May 27, 2016 at 02:42:18PM +0800, Joonsoo Kim wrote:
>> On Fri, May 27, 2016 at 02:25:27PM +0800, Feng Tang wrote:
>>> On Fri, May 27, 2016 at 01:28:20PM +0800, Joonsoo Kim wrote:
 On Thu, May 26, 2016 at 04:04:54PM +0800, Feng Tang wrote:
> On Thu, May 26, 2016 at 02:22:22PM +0800, js1...@gmail.com wrote:
>> From: Joonsoo Kim 
>
>>>  
>> FYI, there is another attempt [3] trying to solve this problem in lkml.
>> And, as far as I know, Qualcomm also has out-of-tree solution for this
>> problem.
>
> This may be a little off-topic :) Actually, we have used another way in
> our products, that we disable the fallback from MIGRATETYE_MOVABLE to
> MIGRATETYPE_CMA completely, and only allow free CMA memory to be used
> by file page cache (which is easy to be reclaimed by its nature). 
> We did it by adding a GFP_PAGE_CACHE to every allocation request for
> page cache, and the MM will try to pick up an available free CMA page
> first, and goes to normal path when fail. 

 Just wonder, why do you allow CMA memory to file page cache rather
 than anonymous page? I guess that anonymous pages would be more easily
 migrated/reclaimed than file page cache. In fact, some of our product
 uses anonymous page adaptation to satisfy similar requirement by
 introducing GFP_CMA. AFAIK, some of chip vendor also uses "anonymous
 page first adaptation" to get better success rate.
>>>
>>> The biggest problem we faced is to allocate big chunk of CMA memory,
>>> say 256MB in a whole, or 9 pieces of 20MB buffers, so the speed
>>> is not the biggest concern, but whether all the cma pages be reclaimed.
>>
>> Okay. Our product have similar workload.
>>
>>> With the MOVABLE fallback, there may be many types of bad guys from device
>>> drivers/kernel or different subsystems, who refuse to return the borrowed
>>> cma pages, so I took a lazy way by only allowing page cache to use free
>>> cma pages, and we see good results which could pass most of the test for
>>> allocating big chunks. 
>>
>> Could you explain more about why file page cache rather than anonymous page?
>> If there is a reason, I'd like to test it by myself.
> 
> I didn't make it clear. This is not for anonymous page, but for 
> MIGRATETYPE_MOVABLE.
> 
> following is the patch to disable the kernel default sharing (kernel 3.14)
> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> index 1b5f20e..a5e698f 100644
> --- a/mm/page_alloc.c
> +++ b/mm/page_alloc.c
> @@ -974,7 +974,11 @@ static int fallbacks[MIGRATE_TYPES][4] = {
>   [MIGRATE_UNMOVABLE]   = { MIGRATE_RECLAIMABLE, MIGRATE_MOVABLE, 
> MIGRATE_RESERVE },
>   [MIGRATE_RECLAIMABLE] = { MIGRATE_UNMOVABLE,   MIGRATE_MOVABLE, 
> MIGRATE_RESERVE },
>  #ifdef CONFIG_CMA
> - [MIGRATE_MOVABLE] = { MIGRATE_CMA, MIGRATE_RECLAIMABLE, 
> MIGRATE_UNMOVABLE, MIGRATE_RESERVE },
> + [MIGRATE_MOVABLE] = { MIGRATE_RECLAIMABLE, MIGRATE_UNMOVABLE, 
> MIGRATE_RESERVE },
>   [MIGRATE_CMA] = { MIGRATE_RESERVE }, /* Never used */
>   [MIGRATE_CMA_ISOLATE] = { MIGRATE_RESERVE }, /* Never used */
>  #else
> @@ -1414,6 +1418,18 @@ void free_hot_cold_page(struct page *page, int cold)
>   local_irq_save(flags);
>   __count_vm_event(PGFREE);
>  
> +#ifndef CONFIG_USE_CMA_FALLBACK
> + if (migratetype == MIGRATE_CMA) {
> + free_one_page(zone, page, 0, MIGRATE_CMA);
> + local_irq_restore(flags);
> + return;
> + }
> +#endif
> +
> 
>>
>>> One of the customer used to use a CMA sharing patch from another vendor
>>> on our Socs, which can't pass these tests and finally took our page cache
>>> approach.
>>
>> CMA has too many problems so each vendor uses their own adaptation. I'd
>> like to solve this code fragmentation by fixing problems on upstream
>> kernel and this ZONE_CMA is one of that effort. If you can share the
>> pointer for your adaptation, it would be very helpful to me.
> 
> As I said, I started to work on CMA problem back in 2014, and faced many
> of these failure in reclamation problems. I didn't have time and capability
> to track/analyze each and every failure, but decided to go another way by
> only allowing the page cache to use CMA.  And frankly speaking, I don't have
> detailed data for performance measurement, but some rough one, that it
> did improve the cma page reclaiming and the usage rate.
> 
> Our patches was based on 3.14 (the Android Mashmallow kenrel). 

Re: [PATCH v3 0/6] Introduce ZONE_CMA

[Chen Feng]

Hi Kim & feng,

Thanks for the share. In our platform also has the same use case.

We only let the alloc with GFP_HIGHUSER_MOVABLE in memory.c to use cma memory.

If we add zone_cma, It seems can resolve the cma migrate issue.

But when free_hot_cold_page, we need let the cma page goto system directly not 
the pcp.
It can be fail while cma_alloc and cma_release. If we alloc the whole cma pages 
which
declared before.

On 2016/5/27 15:27, Feng Tang wrote:
> On Fri, May 27, 2016 at 02:42:18PM +0800, Joonsoo Kim wrote:
>> On Fri, May 27, 2016 at 02:25:27PM +0800, Feng Tang wrote:
>>> On Fri, May 27, 2016 at 01:28:20PM +0800, Joonsoo Kim wrote:
 On Thu, May 26, 2016 at 04:04:54PM +0800, Feng Tang wrote:
> On Thu, May 26, 2016 at 02:22:22PM +0800, js1...@gmail.com wrote:
>> From: Joonsoo Kim 
>
>>>  
>> FYI, there is another attempt [3] trying to solve this problem in lkml.
>> And, as far as I know, Qualcomm also has out-of-tree solution for this
>> problem.
>
> This may be a little off-topic :) Actually, we have used another way in
> our products, that we disable the fallback from MIGRATETYE_MOVABLE to
> MIGRATETYPE_CMA completely, and only allow free CMA memory to be used
> by file page cache (which is easy to be reclaimed by its nature). 
> We did it by adding a GFP_PAGE_CACHE to every allocation request for
> page cache, and the MM will try to pick up an available free CMA page
> first, and goes to normal path when fail. 

 Just wonder, why do you allow CMA memory to file page cache rather
 than anonymous page? I guess that anonymous pages would be more easily
 migrated/reclaimed than file page cache. In fact, some of our product
 uses anonymous page adaptation to satisfy similar requirement by
 introducing GFP_CMA. AFAIK, some of chip vendor also uses "anonymous
 page first adaptation" to get better success rate.
>>>
>>> The biggest problem we faced is to allocate big chunk of CMA memory,
>>> say 256MB in a whole, or 9 pieces of 20MB buffers, so the speed
>>> is not the biggest concern, but whether all the cma pages be reclaimed.
>>
>> Okay. Our product have similar workload.
>>
>>> With the MOVABLE fallback, there may be many types of bad guys from device
>>> drivers/kernel or different subsystems, who refuse to return the borrowed
>>> cma pages, so I took a lazy way by only allowing page cache to use free
>>> cma pages, and we see good results which could pass most of the test for
>>> allocating big chunks. 
>>
>> Could you explain more about why file page cache rather than anonymous page?
>> If there is a reason, I'd like to test it by myself.
> 
> I didn't make it clear. This is not for anonymous page, but for 
> MIGRATETYPE_MOVABLE.
> 
> following is the patch to disable the kernel default sharing (kernel 3.14)
> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> index 1b5f20e..a5e698f 100644
> --- a/mm/page_alloc.c
> +++ b/mm/page_alloc.c
> @@ -974,7 +974,11 @@ static int fallbacks[MIGRATE_TYPES][4] = {
>   [MIGRATE_UNMOVABLE]   = { MIGRATE_RECLAIMABLE, MIGRATE_MOVABLE, 
> MIGRATE_RESERVE },
>   [MIGRATE_RECLAIMABLE] = { MIGRATE_UNMOVABLE,   MIGRATE_MOVABLE, 
> MIGRATE_RESERVE },
>  #ifdef CONFIG_CMA
> - [MIGRATE_MOVABLE] = { MIGRATE_CMA, MIGRATE_RECLAIMABLE, 
> MIGRATE_UNMOVABLE, MIGRATE_RESERVE },
> + [MIGRATE_MOVABLE] = { MIGRATE_RECLAIMABLE, MIGRATE_UNMOVABLE, 
> MIGRATE_RESERVE },
>   [MIGRATE_CMA] = { MIGRATE_RESERVE }, /* Never used */
>   [MIGRATE_CMA_ISOLATE] = { MIGRATE_RESERVE }, /* Never used */
>  #else
> @@ -1414,6 +1418,18 @@ void free_hot_cold_page(struct page *page, int cold)
>   local_irq_save(flags);
>   __count_vm_event(PGFREE);
>  
> +#ifndef CONFIG_USE_CMA_FALLBACK
> + if (migratetype == MIGRATE_CMA) {
> + free_one_page(zone, page, 0, MIGRATE_CMA);
> + local_irq_restore(flags);
> + return;
> + }
> +#endif
> +
> 
>>
>>> One of the customer used to use a CMA sharing patch from another vendor
>>> on our Socs, which can't pass these tests and finally took our page cache
>>> approach.
>>
>> CMA has too many problems so each vendor uses their own adaptation. I'd
>> like to solve this code fragmentation by fixing problems on upstream
>> kernel and this ZONE_CMA is one of that effort. If you can share the
>> pointer for your adaptation, it would be very helpful to me.
> 
> As I said, I started to work on CMA problem back in 2014, and faced many
> of these failure in reclamation problems. I didn't have time and capability
> to track/analyze each and every failure, but decided to go another way by
> only allowing the page cache to use CMA.  And frankly speaking, I don't have
> detailed data for performance measurement, but some rough one, that it
> did improve the cma page reclaiming and the usage rate.
> 
> Our patches was based on 3.14 (the Android Mashmallow kenrel). Earlier this
> year I 

Re: [PATCH v3 0/6] Introduce ZONE_CMA

[Joonsoo Kim]

On Fri, May 27, 2016 at 03:27:02PM +0800, Feng Tang wrote:
> On Fri, May 27, 2016 at 02:42:18PM +0800, Joonsoo Kim wrote:
> > On Fri, May 27, 2016 at 02:25:27PM +0800, Feng Tang wrote:
> > > On Fri, May 27, 2016 at 01:28:20PM +0800, Joonsoo Kim wrote:
> > > > On Thu, May 26, 2016 at 04:04:54PM +0800, Feng Tang wrote:
> > > > > On Thu, May 26, 2016 at 02:22:22PM +0800, js1...@gmail.com wrote:
> > > > > > From: Joonsoo Kim 
> > > > > 
> > >  
> > > > > > FYI, there is another attempt [3] trying to solve this problem in 
> > > > > > lkml.
> > > > > > And, as far as I know, Qualcomm also has out-of-tree solution for 
> > > > > > this
> > > > > > problem.
> > > > > 
> > > > > This may be a little off-topic :) Actually, we have used another way 
> > > > > in
> > > > > our products, that we disable the fallback from MIGRATETYE_MOVABLE to
> > > > > MIGRATETYPE_CMA completely, and only allow free CMA memory to be used
> > > > > by file page cache (which is easy to be reclaimed by its nature). 
> > > > > We did it by adding a GFP_PAGE_CACHE to every allocation request for
> > > > > page cache, and the MM will try to pick up an available free CMA page
> > > > > first, and goes to normal path when fail. 
> > > > 
> > > > Just wonder, why do you allow CMA memory to file page cache rather
> > > > than anonymous page? I guess that anonymous pages would be more easily
> > > > migrated/reclaimed than file page cache. In fact, some of our product
> > > > uses anonymous page adaptation to satisfy similar requirement by
> > > > introducing GFP_CMA. AFAIK, some of chip vendor also uses "anonymous
> > > > page first adaptation" to get better success rate.
> > > 
> > > The biggest problem we faced is to allocate big chunk of CMA memory,
> > > say 256MB in a whole, or 9 pieces of 20MB buffers, so the speed
> > > is not the biggest concern, but whether all the cma pages be reclaimed.
> > 
> > Okay. Our product have similar workload.
> > 
> > > With the MOVABLE fallback, there may be many types of bad guys from device
> > > drivers/kernel or different subsystems, who refuse to return the borrowed
> > > cma pages, so I took a lazy way by only allowing page cache to use free
> > > cma pages, and we see good results which could pass most of the test for
> > > allocating big chunks. 
> > 
> > Could you explain more about why file page cache rather than anonymous page?
> > If there is a reason, I'd like to test it by myself.
> 
> I didn't make it clear. This is not for anonymous page, but for 
> MIGRATETYPE_MOVABLE.

Anonymous page is one of the pages with MIGRATETYPE_MOVABLE. So, you
can also restrict CMA memory only for anonymous page like as you did
for file page cache. Some of our product used this work around so I'd
like to know if there is a reason.

> 
> following is the patch to disable the kernel default sharing (kernel 3.14)
> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> index 1b5f20e..a5e698f 100644
> --- a/mm/page_alloc.c
> +++ b/mm/page_alloc.c
> @@ -974,7 +974,11 @@ static int fallbacks[MIGRATE_TYPES][4] = {
>   [MIGRATE_UNMOVABLE]   = { MIGRATE_RECLAIMABLE, MIGRATE_MOVABLE, 
> MIGRATE_RESERVE },
>   [MIGRATE_RECLAIMABLE] = { MIGRATE_UNMOVABLE,   MIGRATE_MOVABLE, 
> MIGRATE_RESERVE },
>  #ifdef CONFIG_CMA
> - [MIGRATE_MOVABLE] = { MIGRATE_CMA, MIGRATE_RECLAIMABLE, 
> MIGRATE_UNMOVABLE, MIGRATE_RESERVE },
> + [MIGRATE_MOVABLE] = { MIGRATE_RECLAIMABLE, MIGRATE_UNMOVABLE, 
> MIGRATE_RESERVE },
>   [MIGRATE_CMA] = { MIGRATE_RESERVE }, /* Never used */
>   [MIGRATE_CMA_ISOLATE] = { MIGRATE_RESERVE }, /* Never used */
>  #else
> @@ -1414,6 +1418,18 @@ void free_hot_cold_page(struct page *page, int cold)
>   local_irq_save(flags);
>   __count_vm_event(PGFREE);
>  
> +#ifndef CONFIG_USE_CMA_FALLBACK
> + if (migratetype == MIGRATE_CMA) {
> + free_one_page(zone, page, 0, MIGRATE_CMA);
> + local_irq_restore(flags);
> + return;
> + }
> +#endif
> +
> 
> > 
> > > One of the customer used to use a CMA sharing patch from another vendor
> > > on our Socs, which can't pass these tests and finally took our page cache
> > > approach.
> > 
> > CMA has too many problems so each vendor uses their own adaptation. I'd
> > like to solve this code fragmentation by fixing problems on upstream
> > kernel and this ZONE_CMA is one of that effort. If you can share the
> > pointer for your adaptation, it would be very helpful to me.
> 
> As I said, I started to work on CMA problem back in 2014, and faced many
> of these failure in reclamation problems. I didn't have time and capability
> to track/analyze each and every failure, but decided to go another way by
> only allowing the page cache to use CMA.  And frankly speaking, I don't have
> detailed data for performance measurement, but some rough one, that it
> did improve the cma page reclaiming and the usage rate.

Okay!

> Our patches was based on 3.14 (the Android 

Re: [PATCH v3 0/6] Introduce ZONE_CMA

[Joonsoo Kim]

On Fri, May 27, 2016 at 03:27:02PM +0800, Feng Tang wrote:
> On Fri, May 27, 2016 at 02:42:18PM +0800, Joonsoo Kim wrote:
> > On Fri, May 27, 2016 at 02:25:27PM +0800, Feng Tang wrote:
> > > On Fri, May 27, 2016 at 01:28:20PM +0800, Joonsoo Kim wrote:
> > > > On Thu, May 26, 2016 at 04:04:54PM +0800, Feng Tang wrote:
> > > > > On Thu, May 26, 2016 at 02:22:22PM +0800, js1...@gmail.com wrote:
> > > > > > From: Joonsoo Kim 
> > > > > 
> > >  
> > > > > > FYI, there is another attempt [3] trying to solve this problem in 
> > > > > > lkml.
> > > > > > And, as far as I know, Qualcomm also has out-of-tree solution for 
> > > > > > this
> > > > > > problem.
> > > > > 
> > > > > This may be a little off-topic :) Actually, we have used another way 
> > > > > in
> > > > > our products, that we disable the fallback from MIGRATETYE_MOVABLE to
> > > > > MIGRATETYPE_CMA completely, and only allow free CMA memory to be used
> > > > > by file page cache (which is easy to be reclaimed by its nature). 
> > > > > We did it by adding a GFP_PAGE_CACHE to every allocation request for
> > > > > page cache, and the MM will try to pick up an available free CMA page
> > > > > first, and goes to normal path when fail. 
> > > > 
> > > > Just wonder, why do you allow CMA memory to file page cache rather
> > > > than anonymous page? I guess that anonymous pages would be more easily
> > > > migrated/reclaimed than file page cache. In fact, some of our product
> > > > uses anonymous page adaptation to satisfy similar requirement by
> > > > introducing GFP_CMA. AFAIK, some of chip vendor also uses "anonymous
> > > > page first adaptation" to get better success rate.
> > > 
> > > The biggest problem we faced is to allocate big chunk of CMA memory,
> > > say 256MB in a whole, or 9 pieces of 20MB buffers, so the speed
> > > is not the biggest concern, but whether all the cma pages be reclaimed.
> > 
> > Okay. Our product have similar workload.
> > 
> > > With the MOVABLE fallback, there may be many types of bad guys from device
> > > drivers/kernel or different subsystems, who refuse to return the borrowed
> > > cma pages, so I took a lazy way by only allowing page cache to use free
> > > cma pages, and we see good results which could pass most of the test for
> > > allocating big chunks. 
> > 
> > Could you explain more about why file page cache rather than anonymous page?
> > If there is a reason, I'd like to test it by myself.
> 
> I didn't make it clear. This is not for anonymous page, but for 
> MIGRATETYPE_MOVABLE.

Anonymous page is one of the pages with MIGRATETYPE_MOVABLE. So, you
can also restrict CMA memory only for anonymous page like as you did
for file page cache. Some of our product used this work around so I'd
like to know if there is a reason.

> 
> following is the patch to disable the kernel default sharing (kernel 3.14)
> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> index 1b5f20e..a5e698f 100644
> --- a/mm/page_alloc.c
> +++ b/mm/page_alloc.c
> @@ -974,7 +974,11 @@ static int fallbacks[MIGRATE_TYPES][4] = {
>   [MIGRATE_UNMOVABLE]   = { MIGRATE_RECLAIMABLE, MIGRATE_MOVABLE, 
> MIGRATE_RESERVE },
>   [MIGRATE_RECLAIMABLE] = { MIGRATE_UNMOVABLE,   MIGRATE_MOVABLE, 
> MIGRATE_RESERVE },
>  #ifdef CONFIG_CMA
> - [MIGRATE_MOVABLE] = { MIGRATE_CMA, MIGRATE_RECLAIMABLE, 
> MIGRATE_UNMOVABLE, MIGRATE_RESERVE },
> + [MIGRATE_MOVABLE] = { MIGRATE_RECLAIMABLE, MIGRATE_UNMOVABLE, 
> MIGRATE_RESERVE },
>   [MIGRATE_CMA] = { MIGRATE_RESERVE }, /* Never used */
>   [MIGRATE_CMA_ISOLATE] = { MIGRATE_RESERVE }, /* Never used */
>  #else
> @@ -1414,6 +1418,18 @@ void free_hot_cold_page(struct page *page, int cold)
>   local_irq_save(flags);
>   __count_vm_event(PGFREE);
>  
> +#ifndef CONFIG_USE_CMA_FALLBACK
> + if (migratetype == MIGRATE_CMA) {
> + free_one_page(zone, page, 0, MIGRATE_CMA);
> + local_irq_restore(flags);
> + return;
> + }
> +#endif
> +
> 
> > 
> > > One of the customer used to use a CMA sharing patch from another vendor
> > > on our Socs, which can't pass these tests and finally took our page cache
> > > approach.
> > 
> > CMA has too many problems so each vendor uses their own adaptation. I'd
> > like to solve this code fragmentation by fixing problems on upstream
> > kernel and this ZONE_CMA is one of that effort. If you can share the
> > pointer for your adaptation, it would be very helpful to me.
> 
> As I said, I started to work on CMA problem back in 2014, and faced many
> of these failure in reclamation problems. I didn't have time and capability
> to track/analyze each and every failure, but decided to go another way by
> only allowing the page cache to use CMA.  And frankly speaking, I don't have
> detailed data for performance measurement, but some rough one, that it
> did improve the cma page reclaiming and the usage rate.

Okay!

> Our patches was based on 3.14 (the Android Mashmallow kenrel). 

Re: [PATCH v3 0/6] Introduce ZONE_CMA

[Feng Tang]

On Fri, May 27, 2016 at 02:42:18PM +0800, Joonsoo Kim wrote:
> On Fri, May 27, 2016 at 02:25:27PM +0800, Feng Tang wrote:
> > On Fri, May 27, 2016 at 01:28:20PM +0800, Joonsoo Kim wrote:
> > > On Thu, May 26, 2016 at 04:04:54PM +0800, Feng Tang wrote:
> > > > On Thu, May 26, 2016 at 02:22:22PM +0800, js1...@gmail.com wrote:
> > > > > From: Joonsoo Kim 
> > > > 
> >  
> > > > > FYI, there is another attempt [3] trying to solve this problem in 
> > > > > lkml.
> > > > > And, as far as I know, Qualcomm also has out-of-tree solution for this
> > > > > problem.
> > > > 
> > > > This may be a little off-topic :) Actually, we have used another way in
> > > > our products, that we disable the fallback from MIGRATETYE_MOVABLE to
> > > > MIGRATETYPE_CMA completely, and only allow free CMA memory to be used
> > > > by file page cache (which is easy to be reclaimed by its nature). 
> > > > We did it by adding a GFP_PAGE_CACHE to every allocation request for
> > > > page cache, and the MM will try to pick up an available free CMA page
> > > > first, and goes to normal path when fail. 
> > > 
> > > Just wonder, why do you allow CMA memory to file page cache rather
> > > than anonymous page? I guess that anonymous pages would be more easily
> > > migrated/reclaimed than file page cache. In fact, some of our product
> > > uses anonymous page adaptation to satisfy similar requirement by
> > > introducing GFP_CMA. AFAIK, some of chip vendor also uses "anonymous
> > > page first adaptation" to get better success rate.
> > 
> > The biggest problem we faced is to allocate big chunk of CMA memory,
> > say 256MB in a whole, or 9 pieces of 20MB buffers, so the speed
> > is not the biggest concern, but whether all the cma pages be reclaimed.
> 
> Okay. Our product have similar workload.
> 
> > With the MOVABLE fallback, there may be many types of bad guys from device
> > drivers/kernel or different subsystems, who refuse to return the borrowed
> > cma pages, so I took a lazy way by only allowing page cache to use free
> > cma pages, and we see good results which could pass most of the test for
> > allocating big chunks. 
> 
> Could you explain more about why file page cache rather than anonymous page?
> If there is a reason, I'd like to test it by myself.

I didn't make it clear. This is not for anonymous page, but for 
MIGRATETYPE_MOVABLE.

following is the patch to disable the kernel default sharing (kernel 3.14)
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 1b5f20e..a5e698f 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -974,7 +974,11 @@ static int fallbacks[MIGRATE_TYPES][4] = {
[MIGRATE_UNMOVABLE]   = { MIGRATE_RECLAIMABLE, MIGRATE_MOVABLE, 
MIGRATE_RESERVE },
[MIGRATE_RECLAIMABLE] = { MIGRATE_UNMOVABLE,   MIGRATE_MOVABLE, 
MIGRATE_RESERVE },
 #ifdef CONFIG_CMA
-   [MIGRATE_MOVABLE] = { MIGRATE_CMA, MIGRATE_RECLAIMABLE, 
MIGRATE_UNMOVABLE, MIGRATE_RESERVE },
+   [MIGRATE_MOVABLE] = { MIGRATE_RECLAIMABLE, MIGRATE_UNMOVABLE, 
MIGRATE_RESERVE },
[MIGRATE_CMA] = { MIGRATE_RESERVE }, /* Never used */
[MIGRATE_CMA_ISOLATE] = { MIGRATE_RESERVE }, /* Never used */
 #else
@@ -1414,6 +1418,18 @@ void free_hot_cold_page(struct page *page, int cold)
local_irq_save(flags);
__count_vm_event(PGFREE);
 
+#ifndef CONFIG_USE_CMA_FALLBACK
+   if (migratetype == MIGRATE_CMA) {
+   free_one_page(zone, page, 0, MIGRATE_CMA);
+   local_irq_restore(flags);
+   return;
+   }
+#endif
+

> 
> > One of the customer used to use a CMA sharing patch from another vendor
> > on our Socs, which can't pass these tests and finally took our page cache
> > approach.
> 
> CMA has too many problems so each vendor uses their own adaptation. I'd
> like to solve this code fragmentation by fixing problems on upstream
> kernel and this ZONE_CMA is one of that effort. If you can share the
> pointer for your adaptation, it would be very helpful to me.

As I said, I started to work on CMA problem back in 2014, and faced many
of these failure in reclamation problems. I didn't have time and capability
to track/analyze each and every failure, but decided to go another way by
only allowing the page cache to use CMA.  And frankly speaking, I don't have
detailed data for performance measurement, but some rough one, that it
did improve the cma page reclaiming and the usage rate.

Our patches was based on 3.14 (the Android Mashmallow kenrel). Earlier this
year I finally got some free time, and worked on cleaning them for submission
to LKML, and found your cma improving patches merged in 4.1 or 4.2, so I gave
up as my patches is more hacky :)

The sharing patch is here FYI:
--
commit fb28d4db6278df42ab2ef4996bdfd44e613ace99
Author: Feng Tang 
Date:   Wed Jul 15 13:39:50 2015 +0800

cma, page-cache: use cma as page cache

This will free a lot of cma memory for system to use them

Re: [PATCH v3 0/6] Introduce ZONE_CMA

[Feng Tang]

On Fri, May 27, 2016 at 02:42:18PM +0800, Joonsoo Kim wrote:
> On Fri, May 27, 2016 at 02:25:27PM +0800, Feng Tang wrote:
> > On Fri, May 27, 2016 at 01:28:20PM +0800, Joonsoo Kim wrote:
> > > On Thu, May 26, 2016 at 04:04:54PM +0800, Feng Tang wrote:
> > > > On Thu, May 26, 2016 at 02:22:22PM +0800, js1...@gmail.com wrote:
> > > > > From: Joonsoo Kim 
> > > > 
> >  
> > > > > FYI, there is another attempt [3] trying to solve this problem in 
> > > > > lkml.
> > > > > And, as far as I know, Qualcomm also has out-of-tree solution for this
> > > > > problem.
> > > > 
> > > > This may be a little off-topic :) Actually, we have used another way in
> > > > our products, that we disable the fallback from MIGRATETYE_MOVABLE to
> > > > MIGRATETYPE_CMA completely, and only allow free CMA memory to be used
> > > > by file page cache (which is easy to be reclaimed by its nature). 
> > > > We did it by adding a GFP_PAGE_CACHE to every allocation request for
> > > > page cache, and the MM will try to pick up an available free CMA page
> > > > first, and goes to normal path when fail. 
> > > 
> > > Just wonder, why do you allow CMA memory to file page cache rather
> > > than anonymous page? I guess that anonymous pages would be more easily
> > > migrated/reclaimed than file page cache. In fact, some of our product
> > > uses anonymous page adaptation to satisfy similar requirement by
> > > introducing GFP_CMA. AFAIK, some of chip vendor also uses "anonymous
> > > page first adaptation" to get better success rate.
> > 
> > The biggest problem we faced is to allocate big chunk of CMA memory,
> > say 256MB in a whole, or 9 pieces of 20MB buffers, so the speed
> > is not the biggest concern, but whether all the cma pages be reclaimed.
> 
> Okay. Our product have similar workload.
> 
> > With the MOVABLE fallback, there may be many types of bad guys from device
> > drivers/kernel or different subsystems, who refuse to return the borrowed
> > cma pages, so I took a lazy way by only allowing page cache to use free
> > cma pages, and we see good results which could pass most of the test for
> > allocating big chunks. 
> 
> Could you explain more about why file page cache rather than anonymous page?
> If there is a reason, I'd like to test it by myself.

I didn't make it clear. This is not for anonymous page, but for 
MIGRATETYPE_MOVABLE.

following is the patch to disable the kernel default sharing (kernel 3.14)
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 1b5f20e..a5e698f 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -974,7 +974,11 @@ static int fallbacks[MIGRATE_TYPES][4] = {
[MIGRATE_UNMOVABLE]   = { MIGRATE_RECLAIMABLE, MIGRATE_MOVABLE, 
MIGRATE_RESERVE },
[MIGRATE_RECLAIMABLE] = { MIGRATE_UNMOVABLE,   MIGRATE_MOVABLE, 
MIGRATE_RESERVE },
 #ifdef CONFIG_CMA
-   [MIGRATE_MOVABLE] = { MIGRATE_CMA, MIGRATE_RECLAIMABLE, 
MIGRATE_UNMOVABLE, MIGRATE_RESERVE },
+   [MIGRATE_MOVABLE] = { MIGRATE_RECLAIMABLE, MIGRATE_UNMOVABLE, 
MIGRATE_RESERVE },
[MIGRATE_CMA] = { MIGRATE_RESERVE }, /* Never used */
[MIGRATE_CMA_ISOLATE] = { MIGRATE_RESERVE }, /* Never used */
 #else
@@ -1414,6 +1418,18 @@ void free_hot_cold_page(struct page *page, int cold)
local_irq_save(flags);
__count_vm_event(PGFREE);
 
+#ifndef CONFIG_USE_CMA_FALLBACK
+   if (migratetype == MIGRATE_CMA) {
+   free_one_page(zone, page, 0, MIGRATE_CMA);
+   local_irq_restore(flags);
+   return;
+   }
+#endif
+

> 
> > One of the customer used to use a CMA sharing patch from another vendor
> > on our Socs, which can't pass these tests and finally took our page cache
> > approach.
> 
> CMA has too many problems so each vendor uses their own adaptation. I'd
> like to solve this code fragmentation by fixing problems on upstream
> kernel and this ZONE_CMA is one of that effort. If you can share the
> pointer for your adaptation, it would be very helpful to me.

As I said, I started to work on CMA problem back in 2014, and faced many
of these failure in reclamation problems. I didn't have time and capability
to track/analyze each and every failure, but decided to go another way by
only allowing the page cache to use CMA.  And frankly speaking, I don't have
detailed data for performance measurement, but some rough one, that it
did improve the cma page reclaiming and the usage rate.

Our patches was based on 3.14 (the Android Mashmallow kenrel). Earlier this
year I finally got some free time, and worked on cleaning them for submission
to LKML, and found your cma improving patches merged in 4.1 or 4.2, so I gave
up as my patches is more hacky :)

The sharing patch is here FYI:
--
commit fb28d4db6278df42ab2ef4996bdfd44e613ace99
Author: Feng Tang 
Date:   Wed Jul 15 13:39:50 2015 +0800

cma, page-cache: use cma as page cache

This will free a lot of cma memory for system to use them
as page cache. Previously, cma memory is 

Re: [PATCH v3 0/6] Introduce ZONE_CMA

[Joonsoo Kim]

On Fri, May 27, 2016 at 02:25:27PM +0800, Feng Tang wrote:
> On Fri, May 27, 2016 at 01:28:20PM +0800, Joonsoo Kim wrote:
> > On Thu, May 26, 2016 at 04:04:54PM +0800, Feng Tang wrote:
> > > On Thu, May 26, 2016 at 02:22:22PM +0800, js1...@gmail.com wrote:
> > > > From: Joonsoo Kim 
> > > 
>  
> > > > FYI, there is another attempt [3] trying to solve this problem in lkml.
> > > > And, as far as I know, Qualcomm also has out-of-tree solution for this
> > > > problem.
> > > 
> > > This may be a little off-topic :) Actually, we have used another way in
> > > our products, that we disable the fallback from MIGRATETYE_MOVABLE to
> > > MIGRATETYPE_CMA completely, and only allow free CMA memory to be used
> > > by file page cache (which is easy to be reclaimed by its nature). 
> > > We did it by adding a GFP_PAGE_CACHE to every allocation request for
> > > page cache, and the MM will try to pick up an available free CMA page
> > > first, and goes to normal path when fail. 
> > 
> > Just wonder, why do you allow CMA memory to file page cache rather
> > than anonymous page? I guess that anonymous pages would be more easily
> > migrated/reclaimed than file page cache. In fact, some of our product
> > uses anonymous page adaptation to satisfy similar requirement by
> > introducing GFP_CMA. AFAIK, some of chip vendor also uses "anonymous
> > page first adaptation" to get better success rate.
> 
> The biggest problem we faced is to allocate big chunk of CMA memory,
> say 256MB in a whole, or 9 pieces of 20MB buffers, so the speed
> is not the biggest concern, but whether all the cma pages be reclaimed.

Okay. Our product have similar workload.

> With the MOVABLE fallback, there may be many types of bad guys from device
> drivers/kernel or different subsystems, who refuse to return the borrowed
> cma pages, so I took a lazy way by only allowing page cache to use free
> cma pages, and we see good results which could pass most of the test for
> allocating big chunks. 

Could you explain more about why file page cache rather than anonymous page?
If there is a reason, I'd like to test it by myself.

> One of the customer used to use a CMA sharing patch from another vendor
> on our Socs, which can't pass these tests and finally took our page cache
> approach.

CMA has too many problems so each vendor uses their own adaptation. I'd
like to solve this code fragmentation by fixing problems on upstream
kernel and this ZONE_CMA is one of that effort. If you can share the
pointer for your adaptation, it would be very helpful to me.

Thanks.

> > 
> > > It works fine on our products, though we still see some cases that
> > > some page can't be reclaimed. 
> > > 
> > > Our product has a special user case of CMA, that sometimes it will
> > > need to use the whole CMA memory (say 256MB on a phone), then all
> > 
> > I don't think this usecase is so special. Our product also has similar
> > usecase. And, I already knows one another.
> 
> :) I first touch CMA in 2014 and have only worked on Sofia platforms.
> 
> > 
> > > share out CMA pages need to be reclaimed all at once. Don't know if
> > > this new ZONE_CMA approach could meet this request? (our page cache
> > > solution can't ganrantee to meet this request all the time).
> > 
> > This ZONE_CMA approach would be better than before, since CMA memory
> > is not be used for blockdev page cache. Blockdev page cache is one of
> > the frequent failure points in my experience.
> 
> Indeed! I also explicitely disabled cma sharing for blkdev FS page cache.
> 
> > 
> > I'm not sure that ZONE_CMA works better than your GFP_PAGE_CACHE
> > adaptation for your system. In ZONE_CMA, CMA memory is used for file
> > page cache or anonymous pages. If my assumption that anonymous pages
> > are easier to be migrated/reclaimed is correct, ZONE_CMA would work
> > better than your adaptation since there is less file page cache pages
> > in CMA memory.
> > 
> > Anyway, it also doesn't guarantee to succeed all the time. There is
> > different kind of problem that prevents CMA allocation success and we
> > need to solve it. I will try it after problems that this patchset try
> > to fix is solved.
> 
> ZONE_CMA should be cleaner, while our page cache solution needs to
> adjust some policy for lowmemorykiller and page scan/reclaim code.   
> 
> Thanks,
> Feng
> 
> --
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to majord...@kvack.org.  For more info on Linux MM,
> see: http://www.linux-mm.org/ .
> Don't email: mailto:"d...@kvack.org;> em...@kvack.org 

Re: [PATCH v3 0/6] Introduce ZONE_CMA

[Joonsoo Kim]

On Fri, May 27, 2016 at 02:25:27PM +0800, Feng Tang wrote:
> On Fri, May 27, 2016 at 01:28:20PM +0800, Joonsoo Kim wrote:
> > On Thu, May 26, 2016 at 04:04:54PM +0800, Feng Tang wrote:
> > > On Thu, May 26, 2016 at 02:22:22PM +0800, js1...@gmail.com wrote:
> > > > From: Joonsoo Kim 
> > > 
>  
> > > > FYI, there is another attempt [3] trying to solve this problem in lkml.
> > > > And, as far as I know, Qualcomm also has out-of-tree solution for this
> > > > problem.
> > > 
> > > This may be a little off-topic :) Actually, we have used another way in
> > > our products, that we disable the fallback from MIGRATETYE_MOVABLE to
> > > MIGRATETYPE_CMA completely, and only allow free CMA memory to be used
> > > by file page cache (which is easy to be reclaimed by its nature). 
> > > We did it by adding a GFP_PAGE_CACHE to every allocation request for
> > > page cache, and the MM will try to pick up an available free CMA page
> > > first, and goes to normal path when fail. 
> > 
> > Just wonder, why do you allow CMA memory to file page cache rather
> > than anonymous page? I guess that anonymous pages would be more easily
> > migrated/reclaimed than file page cache. In fact, some of our product
> > uses anonymous page adaptation to satisfy similar requirement by
> > introducing GFP_CMA. AFAIK, some of chip vendor also uses "anonymous
> > page first adaptation" to get better success rate.
> 
> The biggest problem we faced is to allocate big chunk of CMA memory,
> say 256MB in a whole, or 9 pieces of 20MB buffers, so the speed
> is not the biggest concern, but whether all the cma pages be reclaimed.

Okay. Our product have similar workload.

> With the MOVABLE fallback, there may be many types of bad guys from device
> drivers/kernel or different subsystems, who refuse to return the borrowed
> cma pages, so I took a lazy way by only allowing page cache to use free
> cma pages, and we see good results which could pass most of the test for
> allocating big chunks. 

Could you explain more about why file page cache rather than anonymous page?
If there is a reason, I'd like to test it by myself.

> One of the customer used to use a CMA sharing patch from another vendor
> on our Socs, which can't pass these tests and finally took our page cache
> approach.

CMA has too many problems so each vendor uses their own adaptation. I'd
like to solve this code fragmentation by fixing problems on upstream
kernel and this ZONE_CMA is one of that effort. If you can share the
pointer for your adaptation, it would be very helpful to me.

Thanks.

> > 
> > > It works fine on our products, though we still see some cases that
> > > some page can't be reclaimed. 
> > > 
> > > Our product has a special user case of CMA, that sometimes it will
> > > need to use the whole CMA memory (say 256MB on a phone), then all
> > 
> > I don't think this usecase is so special. Our product also has similar
> > usecase. And, I already knows one another.
> 
> :) I first touch CMA in 2014 and have only worked on Sofia platforms.
> 
> > 
> > > share out CMA pages need to be reclaimed all at once. Don't know if
> > > this new ZONE_CMA approach could meet this request? (our page cache
> > > solution can't ganrantee to meet this request all the time).
> > 
> > This ZONE_CMA approach would be better than before, since CMA memory
> > is not be used for blockdev page cache. Blockdev page cache is one of
> > the frequent failure points in my experience.
> 
> Indeed! I also explicitely disabled cma sharing for blkdev FS page cache.
> 
> > 
> > I'm not sure that ZONE_CMA works better than your GFP_PAGE_CACHE
> > adaptation for your system. In ZONE_CMA, CMA memory is used for file
> > page cache or anonymous pages. If my assumption that anonymous pages
> > are easier to be migrated/reclaimed is correct, ZONE_CMA would work
> > better than your adaptation since there is less file page cache pages
> > in CMA memory.
> > 
> > Anyway, it also doesn't guarantee to succeed all the time. There is
> > different kind of problem that prevents CMA allocation success and we
> > need to solve it. I will try it after problems that this patchset try
> > to fix is solved.
> 
> ZONE_CMA should be cleaner, while our page cache solution needs to
> adjust some policy for lowmemorykiller and page scan/reclaim code.   
> 
> Thanks,
> Feng
> 
> --
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to majord...@kvack.org.  For more info on Linux MM,
> see: http://www.linux-mm.org/ .
> Don't email: mailto:"d...@kvack.org;> em...@kvack.org 

Re: [PATCH v3 0/6] Introduce ZONE_CMA

[Feng Tang]

On Fri, May 27, 2016 at 01:28:20PM +0800, Joonsoo Kim wrote:
> On Thu, May 26, 2016 at 04:04:54PM +0800, Feng Tang wrote:
> > On Thu, May 26, 2016 at 02:22:22PM +0800, js1...@gmail.com wrote:
> > > From: Joonsoo Kim 
> > 
 
> > > FYI, there is another attempt [3] trying to solve this problem in lkml.
> > > And, as far as I know, Qualcomm also has out-of-tree solution for this
> > > problem.
> > 
> > This may be a little off-topic :) Actually, we have used another way in
> > our products, that we disable the fallback from MIGRATETYE_MOVABLE to
> > MIGRATETYPE_CMA completely, and only allow free CMA memory to be used
> > by file page cache (which is easy to be reclaimed by its nature). 
> > We did it by adding a GFP_PAGE_CACHE to every allocation request for
> > page cache, and the MM will try to pick up an available free CMA page
> > first, and goes to normal path when fail. 
> 
> Just wonder, why do you allow CMA memory to file page cache rather
> than anonymous page? I guess that anonymous pages would be more easily
> migrated/reclaimed than file page cache. In fact, some of our product
> uses anonymous page adaptation to satisfy similar requirement by
> introducing GFP_CMA. AFAIK, some of chip vendor also uses "anonymous
> page first adaptation" to get better success rate.

The biggest problem we faced is to allocate big chunk of CMA memory,
say 256MB in a whole, or 9 pieces of 20MB buffers, so the speed
is not the biggest concern, but whether all the cma pages be reclaimed.

With the MOVABLE fallback, there may be many types of bad guys from device
drivers/kernel or different subsystems, who refuse to return the borrowed
cma pages, so I took a lazy way by only allowing page cache to use free
cma pages, and we see good results which could pass most of the test for
allocating big chunks. 

One of the customer used to use a CMA sharing patch from another vendor
on our Socs, which can't pass these tests and finally took our page cache
approach.

> 
> > It works fine on our products, though we still see some cases that
> > some page can't be reclaimed. 
> > 
> > Our product has a special user case of CMA, that sometimes it will
> > need to use the whole CMA memory (say 256MB on a phone), then all
> 
> I don't think this usecase is so special. Our product also has similar
> usecase. And, I already knows one another.

:) I first touch CMA in 2014 and have only worked on Sofia platforms.

> 
> > share out CMA pages need to be reclaimed all at once. Don't know if
> > this new ZONE_CMA approach could meet this request? (our page cache
> > solution can't ganrantee to meet this request all the time).
> 
> This ZONE_CMA approach would be better than before, since CMA memory
> is not be used for blockdev page cache. Blockdev page cache is one of
> the frequent failure points in my experience.

Indeed! I also explicitely disabled cma sharing for blkdev FS page cache.

> 
> I'm not sure that ZONE_CMA works better than your GFP_PAGE_CACHE
> adaptation for your system. In ZONE_CMA, CMA memory is used for file
> page cache or anonymous pages. If my assumption that anonymous pages
> are easier to be migrated/reclaimed is correct, ZONE_CMA would work
> better than your adaptation since there is less file page cache pages
> in CMA memory.
> 
> Anyway, it also doesn't guarantee to succeed all the time. There is
> different kind of problem that prevents CMA allocation success and we
> need to solve it. I will try it after problems that this patchset try
> to fix is solved.

ZONE_CMA should be cleaner, while our page cache solution needs to
adjust some policy for lowmemorykiller and page scan/reclaim code.   

Thanks,
Feng

Re: [PATCH v3 0/6] Introduce ZONE_CMA

[Feng Tang]

On Fri, May 27, 2016 at 01:28:20PM +0800, Joonsoo Kim wrote:
> On Thu, May 26, 2016 at 04:04:54PM +0800, Feng Tang wrote:
> > On Thu, May 26, 2016 at 02:22:22PM +0800, js1...@gmail.com wrote:
> > > From: Joonsoo Kim 
> > 
 
> > > FYI, there is another attempt [3] trying to solve this problem in lkml.
> > > And, as far as I know, Qualcomm also has out-of-tree solution for this
> > > problem.
> > 
> > This may be a little off-topic :) Actually, we have used another way in
> > our products, that we disable the fallback from MIGRATETYE_MOVABLE to
> > MIGRATETYPE_CMA completely, and only allow free CMA memory to be used
> > by file page cache (which is easy to be reclaimed by its nature). 
> > We did it by adding a GFP_PAGE_CACHE to every allocation request for
> > page cache, and the MM will try to pick up an available free CMA page
> > first, and goes to normal path when fail. 
> 
> Just wonder, why do you allow CMA memory to file page cache rather
> than anonymous page? I guess that anonymous pages would be more easily
> migrated/reclaimed than file page cache. In fact, some of our product
> uses anonymous page adaptation to satisfy similar requirement by
> introducing GFP_CMA. AFAIK, some of chip vendor also uses "anonymous
> page first adaptation" to get better success rate.

The biggest problem we faced is to allocate big chunk of CMA memory,
say 256MB in a whole, or 9 pieces of 20MB buffers, so the speed
is not the biggest concern, but whether all the cma pages be reclaimed.

With the MOVABLE fallback, there may be many types of bad guys from device
drivers/kernel or different subsystems, who refuse to return the borrowed
cma pages, so I took a lazy way by only allowing page cache to use free
cma pages, and we see good results which could pass most of the test for
allocating big chunks. 

One of the customer used to use a CMA sharing patch from another vendor
on our Socs, which can't pass these tests and finally took our page cache
approach.

> 
> > It works fine on our products, though we still see some cases that
> > some page can't be reclaimed. 
> > 
> > Our product has a special user case of CMA, that sometimes it will
> > need to use the whole CMA memory (say 256MB on a phone), then all
> 
> I don't think this usecase is so special. Our product also has similar
> usecase. And, I already knows one another.

:) I first touch CMA in 2014 and have only worked on Sofia platforms.

> 
> > share out CMA pages need to be reclaimed all at once. Don't know if
> > this new ZONE_CMA approach could meet this request? (our page cache
> > solution can't ganrantee to meet this request all the time).
> 
> This ZONE_CMA approach would be better than before, since CMA memory
> is not be used for blockdev page cache. Blockdev page cache is one of
> the frequent failure points in my experience.

Indeed! I also explicitely disabled cma sharing for blkdev FS page cache.

> 
> I'm not sure that ZONE_CMA works better than your GFP_PAGE_CACHE
> adaptation for your system. In ZONE_CMA, CMA memory is used for file
> page cache or anonymous pages. If my assumption that anonymous pages
> are easier to be migrated/reclaimed is correct, ZONE_CMA would work
> better than your adaptation since there is less file page cache pages
> in CMA memory.
> 
> Anyway, it also doesn't guarantee to succeed all the time. There is
> different kind of problem that prevents CMA allocation success and we
> need to solve it. I will try it after problems that this patchset try
> to fix is solved.

ZONE_CMA should be cleaner, while our page cache solution needs to
adjust some policy for lowmemorykiller and page scan/reclaim code.   

Thanks,
Feng

Re: [PATCH v3 0/6] Introduce ZONE_CMA

[Joonsoo Kim]

On Thu, May 26, 2016 at 04:04:54PM +0800, Feng Tang wrote:
> On Thu, May 26, 2016 at 02:22:22PM +0800, js1...@gmail.com wrote:
> > From: Joonsoo Kim 
> 
> Hi Joonsoo,
> 
> Nice work!

Thanks!

> > FYI, there is another attempt [3] trying to solve this problem in lkml.
> > And, as far as I know, Qualcomm also has out-of-tree solution for this
> > problem.
> 
> This may be a little off-topic :) Actually, we have used another way in
> our products, that we disable the fallback from MIGRATETYE_MOVABLE to
> MIGRATETYPE_CMA completely, and only allow free CMA memory to be used
> by file page cache (which is easy to be reclaimed by its nature). 
> We did it by adding a GFP_PAGE_CACHE to every allocation request for
> page cache, and the MM will try to pick up an available free CMA page
> first, and goes to normal path when fail. 

Just wonder, why do you allow CMA memory to file page cache rather
than anonymous page? I guess that anonymous pages would be more easily
migrated/reclaimed than file page cache. In fact, some of our product
uses anonymous page adaptation to satisfy similar requirement by
introducing GFP_CMA. AFAIK, some of chip vendor also uses "anonymous
page first adaptation" to get better success rate.

> It works fine on our products, though we still see some cases that
> some page can't be reclaimed. 
> 
> Our product has a special user case of CMA, that sometimes it will
> need to use the whole CMA memory (say 256MB on a phone), then all

I don't think this usecase is so special. Our product also has similar
usecase. And, I already knows one another.

> share out CMA pages need to be reclaimed all at once. Don't know if
> this new ZONE_CMA approach could meet this request? (our page cache
> solution can't ganrantee to meet this request all the time).

This ZONE_CMA approach would be better than before, since CMA memory
is not be used for blockdev page cache. Blockdev page cache is one of
the frequent failure points in my experience.

I'm not sure that ZONE_CMA works better than your GFP_PAGE_CACHE
adaptation for your system. In ZONE_CMA, CMA memory is used for file
page cache or anonymous pages. If my assumption that anonymous pages
are easier to be migrated/reclaimed is correct, ZONE_CMA would work
better than your adaptation since there is less file page cache pages
in CMA memory.

Anyway, it also doesn't guarantee to succeed all the time. There is
different kind of problem that prevents CMA allocation success and we
need to solve it. I will try it after problems that this patchset try
to fix is solved.

Thanks.

Re: [PATCH v3 0/6] Introduce ZONE_CMA

[Joonsoo Kim]

On Thu, May 26, 2016 at 04:04:54PM +0800, Feng Tang wrote:
> On Thu, May 26, 2016 at 02:22:22PM +0800, js1...@gmail.com wrote:
> > From: Joonsoo Kim 
> 
> Hi Joonsoo,
> 
> Nice work!

Thanks!

> > FYI, there is another attempt [3] trying to solve this problem in lkml.
> > And, as far as I know, Qualcomm also has out-of-tree solution for this
> > problem.
> 
> This may be a little off-topic :) Actually, we have used another way in
> our products, that we disable the fallback from MIGRATETYE_MOVABLE to
> MIGRATETYPE_CMA completely, and only allow free CMA memory to be used
> by file page cache (which is easy to be reclaimed by its nature). 
> We did it by adding a GFP_PAGE_CACHE to every allocation request for
> page cache, and the MM will try to pick up an available free CMA page
> first, and goes to normal path when fail. 

Just wonder, why do you allow CMA memory to file page cache rather
than anonymous page? I guess that anonymous pages would be more easily
migrated/reclaimed than file page cache. In fact, some of our product
uses anonymous page adaptation to satisfy similar requirement by
introducing GFP_CMA. AFAIK, some of chip vendor also uses "anonymous
page first adaptation" to get better success rate.

> It works fine on our products, though we still see some cases that
> some page can't be reclaimed. 
> 
> Our product has a special user case of CMA, that sometimes it will
> need to use the whole CMA memory (say 256MB on a phone), then all

I don't think this usecase is so special. Our product also has similar
usecase. And, I already knows one another.

> share out CMA pages need to be reclaimed all at once. Don't know if
> this new ZONE_CMA approach could meet this request? (our page cache
> solution can't ganrantee to meet this request all the time).

This ZONE_CMA approach would be better than before, since CMA memory
is not be used for blockdev page cache. Blockdev page cache is one of
the frequent failure points in my experience.

I'm not sure that ZONE_CMA works better than your GFP_PAGE_CACHE
adaptation for your system. In ZONE_CMA, CMA memory is used for file
page cache or anonymous pages. If my assumption that anonymous pages
are easier to be migrated/reclaimed is correct, ZONE_CMA would work
better than your adaptation since there is less file page cache pages
in CMA memory.

Anyway, it also doesn't guarantee to succeed all the time. There is
different kind of problem that prevents CMA allocation success and we
need to solve it. I will try it after problems that this patchset try
to fix is solved.

Thanks.

Re: [PATCH v3 0/6] Introduce ZONE_CMA

[Feng Tang]

On Thu, May 26, 2016 at 02:22:22PM +0800, js1...@gmail.com wrote:
> From: Joonsoo Kim 

Hi Joonsoo,

Nice work!

> 
> Hello,
> 
> Changes from v2
> o Rebase on next-20160525
> o No other changes except following description
> 
> There was a discussion with Mel [1] after LSF/MM 2016. I could summarise
> it to help merge decision but it's better to read by yourself since
> if I summarise it, it would be biased for me. But, if anyone hope
> the summary, I will do it. :)
> 
> Anyway, Mel's position on this patchset seems to be neutral. He said:
> "I'm not going to outright NAK your series but I won't ACK it either"
> 
> We can fix the problems with any approach but I hope to go a new zone
> approach because it is less error-prone. It reduces some corner case
> handling for now and remove need for potential corner case handling to fix
> problems.
> 
> Note that our company is already using ZONE_CMA for a years and
> there is no problem.
> 
> If anyone has a different opinion, please let me know and let's discuss
> together.
> 
> Andrew, if there is something to do for merge, please let me know.
> 
> [1] https://lkml.kernel.org/r/20160425053653.GA25662@js1304-P5Q-DELUXE
> 
> Changes from v1
> o Separate some patches which deserve to submit independently
> o Modify description to reflect current kernel state
> (e.g. high-order watermark problem disappeared by Mel's work)
> o Don't increase SECTION_SIZE_BITS to make a room in page flags
> (detailed reason is on the patch that adds ZONE_CMA)
> o Adjust ZONE_CMA population code
> 
> This series try to solve problems of current CMA implementation.
> 
> CMA is introduced to provide physically contiguous pages at runtime
> without exclusive reserved memory area. But, current implementation
> works like as previous reserved memory approach, because freepages
> on CMA region are used only if there is no movable freepage. In other
> words, freepages on CMA region are only used as fallback. In that
> situation where freepages on CMA region are used as fallback, kswapd
> would be woken up easily since there is no unmovable and reclaimable
> freepage, too. If kswapd starts to reclaim memory, fallback allocation
> to MIGRATE_CMA doesn't occur any more since movable freepages are
> already refilled by kswapd and then most of freepage on CMA are left
> to be in free. This situation looks like exclusive reserved memory case.
> 
> In my experiment, I found that if system memory has 1024 MB memory and
> 512 MB is reserved for CMA, kswapd is mostly woken up when roughly 512 MB
> free memory is left. Detailed reason is that for keeping enough free
> memory for unmovable and reclaimable allocation, kswapd uses below
> equation when calculating free memory and it easily go under the watermark.
> 
> Free memory for unmovable and reclaimable = Free total - Free CMA pages
> 
> This is derivated from the property of CMA freepage that CMA freepage
> can't be used for unmovable and reclaimable allocation.
> 
> Anyway, in this case, kswapd are woken up when (FreeTotal - FreeCMA)
> is lower than low watermark and tries to make free memory until
> (FreeTotal - FreeCMA) is higher than high watermark. That results
> in that FreeTotal is moving around 512MB boundary consistently. It
> then means that we can't utilize full memory capacity.
> 
> To fix this problem, I submitted some patches [1] about 10 months ago,
> but, found some more problems to be fixed before solving this problem.
> It requires many hooks in allocator hotpath so some developers doesn't
> like it. Instead, some of them suggest different approach [2] to fix
> all the problems related to CMA, that is, introducing a new zone to deal
> with free CMA pages. I agree that it is the best way to go so implement
> here. Although properties of ZONE_MOVABLE and ZONE_CMA is similar, I
> decide to add a new zone rather than piggyback on ZONE_MOVABLE since
> they have some differences. First, reserved CMA pages should not be
> offlined. If freepage for CMA is managed by ZONE_MOVABLE, we need to keep
> MIGRATE_CMA migratetype and insert many hooks on memory hotplug code
> to distiguish hotpluggable memory and reserved memory for CMA in the same
> zone. It would make memory hotplug code which is already complicated
> more complicated. Second, cma_alloc() can be called more frequently
> than memory hotplug operation and possibly we need to control
> allocation rate of ZONE_CMA to optimize latency in the future.
> In this case, separate zone approach is easy to modify. Third, I'd
> like to see statistics for CMA, separately. Sometimes, we need to debug
> why cma_alloc() is failed and separate statistics would be more helpful
> in this situtaion.
> 
> Anyway, this patchset solves four problems related to CMA implementation.
> 
> 1) Utilization problem
> As mentioned above, we can't utilize full memory capacity due to the
> limitation of CMA freepage and fallback policy. This patchset implements
> a new zone for CMA and 

Re: [PATCH v3 0/6] Introduce ZONE_CMA

[Feng Tang]

On Thu, May 26, 2016 at 02:22:22PM +0800, js1...@gmail.com wrote:
> From: Joonsoo Kim 

Hi Joonsoo,

Nice work!

> 
> Hello,
> 
> Changes from v2
> o Rebase on next-20160525
> o No other changes except following description
> 
> There was a discussion with Mel [1] after LSF/MM 2016. I could summarise
> it to help merge decision but it's better to read by yourself since
> if I summarise it, it would be biased for me. But, if anyone hope
> the summary, I will do it. :)
> 
> Anyway, Mel's position on this patchset seems to be neutral. He said:
> "I'm not going to outright NAK your series but I won't ACK it either"
> 
> We can fix the problems with any approach but I hope to go a new zone
> approach because it is less error-prone. It reduces some corner case
> handling for now and remove need for potential corner case handling to fix
> problems.
> 
> Note that our company is already using ZONE_CMA for a years and
> there is no problem.
> 
> If anyone has a different opinion, please let me know and let's discuss
> together.
> 
> Andrew, if there is something to do for merge, please let me know.
> 
> [1] https://lkml.kernel.org/r/20160425053653.GA25662@js1304-P5Q-DELUXE
> 
> Changes from v1
> o Separate some patches which deserve to submit independently
> o Modify description to reflect current kernel state
> (e.g. high-order watermark problem disappeared by Mel's work)
> o Don't increase SECTION_SIZE_BITS to make a room in page flags
> (detailed reason is on the patch that adds ZONE_CMA)
> o Adjust ZONE_CMA population code
> 
> This series try to solve problems of current CMA implementation.
> 
> CMA is introduced to provide physically contiguous pages at runtime
> without exclusive reserved memory area. But, current implementation
> works like as previous reserved memory approach, because freepages
> on CMA region are used only if there is no movable freepage. In other
> words, freepages on CMA region are only used as fallback. In that
> situation where freepages on CMA region are used as fallback, kswapd
> would be woken up easily since there is no unmovable and reclaimable
> freepage, too. If kswapd starts to reclaim memory, fallback allocation
> to MIGRATE_CMA doesn't occur any more since movable freepages are
> already refilled by kswapd and then most of freepage on CMA are left
> to be in free. This situation looks like exclusive reserved memory case.
> 
> In my experiment, I found that if system memory has 1024 MB memory and
> 512 MB is reserved for CMA, kswapd is mostly woken up when roughly 512 MB
> free memory is left. Detailed reason is that for keeping enough free
> memory for unmovable and reclaimable allocation, kswapd uses below
> equation when calculating free memory and it easily go under the watermark.
> 
> Free memory for unmovable and reclaimable = Free total - Free CMA pages
> 
> This is derivated from the property of CMA freepage that CMA freepage
> can't be used for unmovable and reclaimable allocation.
> 
> Anyway, in this case, kswapd are woken up when (FreeTotal - FreeCMA)
> is lower than low watermark and tries to make free memory until
> (FreeTotal - FreeCMA) is higher than high watermark. That results
> in that FreeTotal is moving around 512MB boundary consistently. It
> then means that we can't utilize full memory capacity.
> 
> To fix this problem, I submitted some patches [1] about 10 months ago,
> but, found some more problems to be fixed before solving this problem.
> It requires many hooks in allocator hotpath so some developers doesn't
> like it. Instead, some of them suggest different approach [2] to fix
> all the problems related to CMA, that is, introducing a new zone to deal
> with free CMA pages. I agree that it is the best way to go so implement
> here. Although properties of ZONE_MOVABLE and ZONE_CMA is similar, I
> decide to add a new zone rather than piggyback on ZONE_MOVABLE since
> they have some differences. First, reserved CMA pages should not be
> offlined. If freepage for CMA is managed by ZONE_MOVABLE, we need to keep
> MIGRATE_CMA migratetype and insert many hooks on memory hotplug code
> to distiguish hotpluggable memory and reserved memory for CMA in the same
> zone. It would make memory hotplug code which is already complicated
> more complicated. Second, cma_alloc() can be called more frequently
> than memory hotplug operation and possibly we need to control
> allocation rate of ZONE_CMA to optimize latency in the future.
> In this case, separate zone approach is easy to modify. Third, I'd
> like to see statistics for CMA, separately. Sometimes, we need to debug
> why cma_alloc() is failed and separate statistics would be more helpful
> in this situtaion.
> 
> Anyway, this patchset solves four problems related to CMA implementation.
> 
> 1) Utilization problem
> As mentioned above, we can't utilize full memory capacity due to the
> limitation of CMA freepage and fallback policy. This patchset implements
> a new zone for CMA and uses it for 

[PATCH v3 0/6] Introduce ZONE_CMA

[js1304]

From: Joonsoo Kim 

Hello,

Changes from v2
o Rebase on next-20160525
o No other changes except following description

There was a discussion with Mel [1] after LSF/MM 2016. I could summarise
it to help merge decision but it's better to read by yourself since
if I summarise it, it would be biased for me. But, if anyone hope
the summary, I will do it. :)

Anyway, Mel's position on this patchset seems to be neutral. He said:
"I'm not going to outright NAK your series but I won't ACK it either"

We can fix the problems with any approach but I hope to go a new zone
approach because it is less error-prone. It reduces some corner case
handling for now and remove need for potential corner case handling to fix
problems.

Note that our company is already using ZONE_CMA for a years and
there is no problem.

If anyone has a different opinion, please let me know and let's discuss
together.

Andrew, if there is something to do for merge, please let me know.

[1] https://lkml.kernel.org/r/20160425053653.GA25662@js1304-P5Q-DELUXE

Changes from v1
o Separate some patches which deserve to submit independently
o Modify description to reflect current kernel state
(e.g. high-order watermark problem disappeared by Mel's work)
o Don't increase SECTION_SIZE_BITS to make a room in page flags
(detailed reason is on the patch that adds ZONE_CMA)
o Adjust ZONE_CMA population code

This series try to solve problems of current CMA implementation.

CMA is introduced to provide physically contiguous pages at runtime
without exclusive reserved memory area. But, current implementation
works like as previous reserved memory approach, because freepages
on CMA region are used only if there is no movable freepage. In other
words, freepages on CMA region are only used as fallback. In that
situation where freepages on CMA region are used as fallback, kswapd
would be woken up easily since there is no unmovable and reclaimable
freepage, too. If kswapd starts to reclaim memory, fallback allocation
to MIGRATE_CMA doesn't occur any more since movable freepages are
already refilled by kswapd and then most of freepage on CMA are left
to be in free. This situation looks like exclusive reserved memory case.

In my experiment, I found that if system memory has 1024 MB memory and
512 MB is reserved for CMA, kswapd is mostly woken up when roughly 512 MB
free memory is left. Detailed reason is that for keeping enough free
memory for unmovable and reclaimable allocation, kswapd uses below
equation when calculating free memory and it easily go under the watermark.

Free memory for unmovable and reclaimable = Free total - Free CMA pages

This is derivated from the property of CMA freepage that CMA freepage
can't be used for unmovable and reclaimable allocation.

Anyway, in this case, kswapd are woken up when (FreeTotal - FreeCMA)
is lower than low watermark and tries to make free memory until
(FreeTotal - FreeCMA) is higher than high watermark. That results
in that FreeTotal is moving around 512MB boundary consistently. It
then means that we can't utilize full memory capacity.

To fix this problem, I submitted some patches [1] about 10 months ago,
but, found some more problems to be fixed before solving this problem.
It requires many hooks in allocator hotpath so some developers doesn't
like it. Instead, some of them suggest different approach [2] to fix
all the problems related to CMA, that is, introducing a new zone to deal
with free CMA pages. I agree that it is the best way to go so implement
here. Although properties of ZONE_MOVABLE and ZONE_CMA is similar, I
decide to add a new zone rather than piggyback on ZONE_MOVABLE since
they have some differences. First, reserved CMA pages should not be
offlined. If freepage for CMA is managed by ZONE_MOVABLE, we need to keep
MIGRATE_CMA migratetype and insert many hooks on memory hotplug code
to distiguish hotpluggable memory and reserved memory for CMA in the same
zone. It would make memory hotplug code which is already complicated
more complicated. Second, cma_alloc() can be called more frequently
than memory hotplug operation and possibly we need to control
allocation rate of ZONE_CMA to optimize latency in the future.
In this case, separate zone approach is easy to modify. Third, I'd
like to see statistics for CMA, separately. Sometimes, we need to debug
why cma_alloc() is failed and separate statistics would be more helpful
in this situtaion.

Anyway, this patchset solves four problems related to CMA implementation.

1) Utilization problem
As mentioned above, we can't utilize full memory capacity due to the
limitation of CMA freepage and fallback policy. This patchset implements
a new zone for CMA and uses it for GFP_HIGHUSER_MOVABLE request. This
typed allocation is used for page cache and anonymous pages which
occupies most of memory usage in normal case so we can utilize full
memory capacity. Below is the experiment result about this problem.

8 CPUs, 1024 MB, VIRTUAL MACHINE
make 

[PATCH v3 0/6] Introduce ZONE_CMA

[js1304]

From: Joonsoo Kim 

Hello,

Changes from v2
o Rebase on next-20160525
o No other changes except following description

There was a discussion with Mel [1] after LSF/MM 2016. I could summarise
it to help merge decision but it's better to read by yourself since
if I summarise it, it would be biased for me. But, if anyone hope
the summary, I will do it. :)

Anyway, Mel's position on this patchset seems to be neutral. He said:
"I'm not going to outright NAK your series but I won't ACK it either"

We can fix the problems with any approach but I hope to go a new zone
approach because it is less error-prone. It reduces some corner case
handling for now and remove need for potential corner case handling to fix
problems.

Note that our company is already using ZONE_CMA for a years and
there is no problem.

If anyone has a different opinion, please let me know and let's discuss
together.

Andrew, if there is something to do for merge, please let me know.

[1] https://lkml.kernel.org/r/20160425053653.GA25662@js1304-P5Q-DELUXE

Changes from v1
o Separate some patches which deserve to submit independently
o Modify description to reflect current kernel state
(e.g. high-order watermark problem disappeared by Mel's work)
o Don't increase SECTION_SIZE_BITS to make a room in page flags
(detailed reason is on the patch that adds ZONE_CMA)
o Adjust ZONE_CMA population code

This series try to solve problems of current CMA implementation.

CMA is introduced to provide physically contiguous pages at runtime
without exclusive reserved memory area. But, current implementation
works like as previous reserved memory approach, because freepages
on CMA region are used only if there is no movable freepage. In other
words, freepages on CMA region are only used as fallback. In that
situation where freepages on CMA region are used as fallback, kswapd
would be woken up easily since there is no unmovable and reclaimable
freepage, too. If kswapd starts to reclaim memory, fallback allocation
to MIGRATE_CMA doesn't occur any more since movable freepages are
already refilled by kswapd and then most of freepage on CMA are left
to be in free. This situation looks like exclusive reserved memory case.

In my experiment, I found that if system memory has 1024 MB memory and
512 MB is reserved for CMA, kswapd is mostly woken up when roughly 512 MB
free memory is left. Detailed reason is that for keeping enough free
memory for unmovable and reclaimable allocation, kswapd uses below
equation when calculating free memory and it easily go under the watermark.

Free memory for unmovable and reclaimable = Free total - Free CMA pages

This is derivated from the property of CMA freepage that CMA freepage
can't be used for unmovable and reclaimable allocation.

Anyway, in this case, kswapd are woken up when (FreeTotal - FreeCMA)
is lower than low watermark and tries to make free memory until
(FreeTotal - FreeCMA) is higher than high watermark. That results
in that FreeTotal is moving around 512MB boundary consistently. It
then means that we can't utilize full memory capacity.

To fix this problem, I submitted some patches [1] about 10 months ago,
but, found some more problems to be fixed before solving this problem.
It requires many hooks in allocator hotpath so some developers doesn't
like it. Instead, some of them suggest different approach [2] to fix
all the problems related to CMA, that is, introducing a new zone to deal
with free CMA pages. I agree that it is the best way to go so implement
here. Although properties of ZONE_MOVABLE and ZONE_CMA is similar, I
decide to add a new zone rather than piggyback on ZONE_MOVABLE since
they have some differences. First, reserved CMA pages should not be
offlined. If freepage for CMA is managed by ZONE_MOVABLE, we need to keep
MIGRATE_CMA migratetype and insert many hooks on memory hotplug code
to distiguish hotpluggable memory and reserved memory for CMA in the same
zone. It would make memory hotplug code which is already complicated
more complicated. Second, cma_alloc() can be called more frequently
than memory hotplug operation and possibly we need to control
allocation rate of ZONE_CMA to optimize latency in the future.
In this case, separate zone approach is easy to modify. Third, I'd
like to see statistics for CMA, separately. Sometimes, we need to debug
why cma_alloc() is failed and separate statistics would be more helpful
in this situtaion.

Anyway, this patchset solves four problems related to CMA implementation.

1) Utilization problem
As mentioned above, we can't utilize full memory capacity due to the
limitation of CMA freepage and fallback policy. This patchset implements
a new zone for CMA and uses it for GFP_HIGHUSER_MOVABLE request. This
typed allocation is used for page cache and anonymous pages which
occupies most of memory usage in normal case so we can utilize full
memory capacity. Below is the experiment result about this problem.

8 CPUs, 1024 MB, VIRTUAL MACHINE
make -j16


CMA reserve: