Re: [dpdk-users] Memory allocation limits

[Dave Sclarsky]

I apologize, I misread some numbers - I'm actually getting 128GB per numa!
I would still like to increase to get ~192GB per numa if possible!
BTW, I forgot to mention that I'm using DPDK 21.11...
Thanks for any advice!

Re: [dpdk-users] Memory allocation limits

[Dave Sclarsky]

Hi, I'm have a similar problem (but with a dell server with 512GB memory, 256 
in each numa).
I can only allocate 128GB, 64 from each NUMA.  If I try 130GB (65 from each 
numa), I get "eal_memalloc_alloc_seg_bulk(): couldn't find suitable 
memseg_list".
I tried changing the config variables mentioned in "3.1.4.5 Maximum Amount of 
Memory" , but no matter what I do to the RTE_MAX_MEMxxx values in rte_config.h 
the limit seems to be the same.  I tried multiplying all of them by 
4,8,16...nothing helped. I also see that RTE_MAX_MEM_MB is "hard coded" to 
524288 for ARCH_64 in meson.build, which should be enough so I didn't try to 
change that one.

Has anyone succeeded in allocating more than 128GB?

Re: [dpdk-users] Memory allocation limits

[Thomas Monjalon]

29/09/2021 14:43, Burakov, Anatoly:
> From: Thomas Monjalon 
> > 29/09/2021 12:14, Burakov, Anatoly:
> > > From: Thomas Monjalon 
> > > > 26/09/2021 17:52, Mohammad Masumi:
> > > > > I have HP server with 768GB memory 384GB in each Numa but I can't
> > > > > allocate more than 64GB by rte_malloc by changing some parameters
> > > > > in rte_config.h it increased to 128GB How to increase heap size?
> > >
> > > This is intentional. In order to increase the amount of contiguous 
> > > allocation
> > possible to perform in DPDK, you need to adjust the following values in
> > rte_config.h:
> > >
> > > #define RTE_MAX_MEMSEG_PER_LIST 8192
> > > #define RTE_MAX_MEM_MB_PER_LIST 32768
> > > #define RTE_MAX_MEMSEG_PER_TYPE 32768
> > > #define RTE_MAX_MEM_MB_PER_TYPE 65536
> > >
> > > I do not recommend arbitrarily changing them as this is untested, but
> > increasing them proportionally (e.g. multiply all of them by 2 or 4) should 
> > not
> > break anything.
> > 
> > It looks to be something to add in docs, right?
> 
> [[AB]] 
> Yes, which is why I already have 
> 
> http://doc.dpdk.org/guides/prog_guide/env_abstraction_layer.html#memory-mapping-discovery-and-memory-reservation
> 
> There's a section on "Maximum amount of memory" there.

It says "Normally, these options do not need to be changed."
Would it be meaningful to add how we can increase (by multiplying all of them)?

RE: [dpdk-users] Memory allocation limits

[Burakov, Anatoly]

> -Original Message-
> From: Thomas Monjalon 
> Sent: Wednesday, September 29, 2021 12:12 PM
> To: Mohammad Masumi ; Burakov, Anatoly
> 
> Cc: users@dpdk.org; dmitry.kozl...@gmail.com; Richardson, Bruce
> ; step...@networkplumber.org
> Subject: Re: [dpdk-users] Memory allocation limits
> 
> 29/09/2021 12:14, Burakov, Anatoly:
> > From: Thomas Monjalon 
> > > 26/09/2021 17:52, Mohammad Masumi:
> > > > Hi
> > > >
> > > > I have HP server with 768GB memory 384GB in each Numa but I can't
> > > > allocate more than 64GB by rte_malloc by changing some parameters
> > > > in rte_config.h it increased to 128GB How to increase heap size?
> > >
> > > adding people Cc to help
> > >
> >
> > Hi,
> >
> > This is intentional. In order to increase the amount of contiguous 
> > allocation
> possible to perform in DPDK, you need to adjust the following values in
> rte_config.h:
> >
> > #define RTE_MAX_MEMSEG_PER_LIST 8192
> > #define RTE_MAX_MEM_MB_PER_LIST 32768
> > #define RTE_MAX_MEMSEG_PER_TYPE 32768
> > #define RTE_MAX_MEM_MB_PER_TYPE 65536
> >
> > I do not recommend arbitrarily changing them as this is untested, but
> increasing them proportionally (e.g. multiply all of them by 2 or 4) should 
> not
> break anything.
> 
> It looks to be something to add in docs, right?
> 

[[AB]] 
Yes, which is why I already have 

http://doc.dpdk.org/guides/prog_guide/env_abstraction_layer.html#memory-mapping-discovery-and-memory-reservation

There's a section on "Maximum amount of memory" there.

Re: [dpdk-users] Memory allocation limits

[Thomas Monjalon]

29/09/2021 12:14, Burakov, Anatoly:
> From: Thomas Monjalon 
> > 26/09/2021 17:52, Mohammad Masumi:
> > > Hi
> > >
> > > I have HP server with 768GB memory 384GB in each Numa but I can't
> > > allocate more than 64GB by rte_malloc by changing some parameters in
> > > rte_config.h it increased to 128GB How to increase heap size?
> > 
> > adding people Cc to help
> > 
> 
> Hi,
> 
> This is intentional. In order to increase the amount of contiguous allocation 
> possible to perform in DPDK, you need to adjust the following values in 
> rte_config.h:
> 
> #define RTE_MAX_MEMSEG_PER_LIST 8192
> #define RTE_MAX_MEM_MB_PER_LIST 32768
> #define RTE_MAX_MEMSEG_PER_TYPE 32768
> #define RTE_MAX_MEM_MB_PER_TYPE 65536
> 
> I do not recommend arbitrarily changing them as this is untested, but 
> increasing them proportionally (e.g. multiply all of them by 2 or 4) should 
> not break anything.

It looks to be something to add in docs, right?

RE: [dpdk-users] Memory allocation limits

[Burakov, Anatoly]

> -Original Message-
> From: Thomas Monjalon 
> Sent: Wednesday, September 29, 2021 10:59 AM
> To: Mohammad Masumi 
> Cc: users@dpdk.org; Burakov, Anatoly ;
> dmitry.kozl...@gmail.com; Richardson, Bruce
> ; step...@networkplumber.org
> Subject: Re: [dpdk-users] Memory allocation limits
> 
> 26/09/2021 17:52, Mohammad Masumi:
> > Hi
> >
> > I have HP server with 768GB memory 384GB in each Numa but I can't
> > allocate more than 64GB by rte_malloc by changing some parameters in
> > rte_config.h it increased to 128GB How to increase heap size?
> 
> adding people Cc to help
> 

Hi,

This is intentional. In order to increase the amount of contiguous allocation 
possible to perform in DPDK, you need to adjust the following values in 
rte_config.h:

#define RTE_MAX_MEMSEG_PER_LIST 8192
#define RTE_MAX_MEM_MB_PER_LIST 32768
#define RTE_MAX_MEMSEG_PER_TYPE 32768
#define RTE_MAX_MEM_MB_PER_TYPE 65536

I do not recommend arbitrarily changing them as this is untested, but 
increasing them proportionally (e.g. multiply all of them by 2 or 4) should not 
break anything.

Thanks,
Anatoly

Re: [dpdk-users] Memory allocation limits

[Thomas Monjalon]

26/09/2021 17:52, Mohammad Masumi:
> Hi
> 
> I have HP server with 768GB memory 384GB in each Numa but I can't allocate
> more than 64GB by rte_malloc by changing some parameters in rte_config.h it
> increased to 128GB
> How to increase heap size?

adding people Cc to help

[dpdk-users] Memory allocation limits

[Mohammad Masumi]

Hi

I have HP server with 768GB memory 384GB in each Numa but I can't allocate
more than 64GB by rte_malloc by changing some parameters in rte_config.h it
increased to 128GB
How to increase heap size?

[dpdk-users] Memory allocation issue after upgrading to DPDK18.11

[Ram, Hila]

Hi,
It's my first time upgrading DPDK version (from DPDK16.07 to DPDK18.11.5).
Some setup relevant info:
* Two Numas.
* Single page size of 1G.
* Total of 95 huge pages, 48 are being allocated for socket0 and 47 for 
socket1.
After upgrading to 18.11, I get failure on application startup while trying to 
alloc some memory block, which is greater than Greatest_free_size and no other 
pages are left. With the same application and amount of huge pages, these 
allocations ended successfully when running with DPDK16.07.
So I compared the allocation on 16.07 VS 18.11 and found out that when memory 
is allocated statically in 16.07 we have no issue (where Free_size always 
equals to Greatest_free_size), but when allocating memory dynamically we 
allocate a new page whenever there is no enough contiguous memory, so I guess 
we have some holes and as a result no free memory is left.
 1. When working dynamically, doesn't the DPDK try to allocate 
contiguously? If it does, why are we out of memory?
In order to overcome it I enabled the memory legacy flag. However, once using 
it, I get the following error - EAL: Could not find space for memseg. Please 
increase CONFIG_RTE_MAX_MEMSEG_PER_TYPE and/or CONFIG_RTE_MAX_MEM_PER_TYPE in 
configuration.
With default configuration, each segment list has 32 segments (as being 
determined by memseg_primary_init), 4 lists per socket.
When we get to remap_segment (getting there from eal_legacy_hugepage_init  -> 
remap_needed_hugepages), and go through each memseg list searching for n=48 
free segments. However, on each list we have only 32 segments so we fail to 
remap segments.
As a workaround I increased CONFIG_RTE_MAX_MEM_MB_PER_LIST from 32768 to 65536 
so as a result each segment list holds 64 segments. Modifying 
CONFIG_RTE_MAX_MEMSEG_PER_TYPE and/or CONFIG_RTE_MAX_MEM_PER_TYPE as printed in 
error message had no influence as calculation takes into account the min value 
of per_type and per_list, where per list was the smaller one.
 2. Why do we check for n segments in a single list instead of in all lists?
 3. Is it correct handling increasing num of segments per list? What if we 
have more huge pages (which result with more than 64 per socket) - we should 
re-increase it?
Thanks,
Hila




This electronic message may contain proprietary and confidential information of 
Verint Systems Inc., its affiliates and/or subsidiaries. The information is 
intended to be for the use of the individual(s) or entity(ies) named above. If 
you are not the intended recipient (or authorized to receive this e-mail for 
the intended recipient), you may not use, copy, disclose or distribute to 
anyone this message or any information contained in this message. If you have 
received this electronic message in error, please notify us by replying to this 
e-mail.

Re: [dpdk-users] Memory allocation in dpdk

[Wiles, Keith]

> On Oct 23, 2018, at 1:44 AM, Avinash Chaurasia  
> wrote:
> 
> Hello,
> I am trying to understand how dpdk allocate memory. I tried digging code to
> understand memory allocation of DPDK. So far I understood that memory is
> allocated from a heap that dpdk maintains. However, this heap must be
> allocated at some place. I failed to traceback any function (called from
> heap_alloc()) that calls mmap to allocate memory. Please let me know when
> this heap is created, which function call does that.

Memory allocation is tricky in DPDK, but are you talking about rte_malloc or 
rte_mempool allocagtion. Each h has a different way to get memory. Look at the 
rte_memzone code it does the lowest level allocation of memory and use huge 
pages. Also look at the email list for patches submitted by Anatoly Burakov he 
just re-wrote the memory system and has some good explanations in the patches 
an in the docs for DPDK.

> Thanks
> Avinash Kumar Chaurasia

Regards,
Keith

[dpdk-users] Memory allocation in dpdk

[Avinash Chaurasia]

Hello,
I am trying to understand how dpdk allocate memory. I tried digging code to
understand memory allocation of DPDK. So far I understood that memory is
allocated from a heap that dpdk maintains. However, this heap must be
allocated at some place. I failed to traceback any function (called from
heap_alloc()) that calls mmap to allocate memory. Please let me know when
this heap is created, which function call does that.
Thanks
Avinash Kumar Chaurasia

Re: [dpdk-users] Memory Allocation

[Anupam Kapoor]

On Tue, May 16, 2017 at 9:44 PM, Kumaraparameshwaran Rathnavel <
kr...@cloudsimple.com> wrote:

> will the memory be allocated on the Node where the DPDK thread was
> initially running when the memory allocation functions was called


​rte_malloc​(...) will return memory allocated on the NUMA socket of the
core which made the invokation. access to this from other locations will
incur the qpi penalty...

​--
thanks
anupam​


In the beginning was the lambda, and the lambda was with Emacs, and Emacs
was the lambda.

Re: [dpdk-users] Memory Allocation

[Stephen Hemminger]

On Tue, 16 May 2017 21:44:46 +0530
Kumaraparameshwaran Rathnavel  wrote:

> Hi All,
> 
> When I have a system with NUMA enabled and say my application is running one 
> DPDK thread and I use that thread for the memory allocation will the memory 
> be physically allocated during the access time or will the memory be 
> allocated on the Node where the DPDK thread was initially running when the 
> memory allocation functions was called. So threads accessing the memory 
> locations from the other nodes remotely will suffer a degradation in 
> performance. Please correct me if I am wrong. 
> 
> Thanking You,
> Param.
> 
> 

Most of the memory allocation mechansisms (malloc, mmap) in Linux are not NUMA 
aware.
If you read the source (which is always the definitive place to look), for DPDK 
you
will see that rte_malloc, rte_mempool, and other routines have option to take a 
NUMA
node as argument. It is up to the application to determine what the "right" 
node is.
Often it is the node where the hardware is located (for mbuf pools), or the node
where the thread is running. But unless you lock the thread to a node, the 
scheduler
is likely to move it around under load.