> -----Original Message----- > From: Bill Fischofer [mailto:bill.fischo...@linaro.org] > Sent: Tuesday, December 13, 2016 4:00 PM > To: Savolainen, Petri (Nokia - FI/Espoo) <petri.savolainen@nokia-bell- > labs.com> > Cc: Maxim Uvarov <maxim.uva...@linaro.org>; lng-odp-forward <lng- > o...@lists.linaro.org> > Subject: Re: [lng-odp] sizeof(pool_table_t) = 272M > > On Tue, Dec 13, 2016 at 3:23 AM, Savolainen, Petri (Nokia - FI/Espoo) > <petri.savolai...@nokia-bell-labs.com> wrote: > > > > > >> -----Original Message----- > >> From: lng-odp [mailto:lng-odp-boun...@lists.linaro.org] On Behalf Of > Maxim > >> Uvarov > >> Sent: Monday, December 12, 2016 6:45 PM > >> To: lng-odp-forward <lng-odp@lists.linaro.org> > >> Subject: [lng-odp] sizeof(pool_table_t) = 272M > >> > >> The latest pool changed set huge pool_table_t. That is due to: > >> > >> #define CONFIG_POOL_MAX_NUM (1 * 1024 * 1024) > >> > >> If for example it's > >> #define CONFIG_POOL_MAX_NUM (4096) > >> > >> Then pool_table_t is 18M. > >> > >> > >> Allocation happens here: > >> int odp_pool_init_global(void) > >> { > >> > >> shm = odp_shm_reserve("_odp_pool_table", > >> sizeof(pool_table_t), > >> ODP_CACHE_LINE_SIZE, 0); > >> > >> ... > >> > >> > >> Because of with latest changes in api-next for process mode and drivers > >> mapping file in /tmp/ directory is created. So for each process there > is > >> 272M file. I think that is too big and can be a problem with testing on > >> small machines. > >> > >> 1. Can we lower that number of maximum entries in the pool? > > > > > > This is why those are config options. We can trade-off capability to > memory usage. Both of these affect to the memory usage: > > > > /* > > * Maximum number of pools > > */ > > #define ODP_CONFIG_POOLS 64 > > > > /* > > * Maximum number of events in a pool > > */ > > #define CONFIG_POOL_MAX_NUM (1 * 1024 * 1024) > > > > > > If you cut both in half, the memory usage drops into one fourth... > > > > > >> > >> 2. Can we remove that table from here completely and made it dynamic in > >> odp_pool_create()? > > > > > > Not really, it would be unnecessary complicated compared to the gain. > For example, I have 8GB memory on my laptop, 128 GB on my server and even > raspberry pi has 1 GB. Laptop would run one/two instances, server a few > (up to 6), raspberry only one. > > > > The current memory consumption would be problem only for a raspberry > user, but it would be very easy for him/her to scale down this (and other) > config options to match the system resources. Default ODP configuration is > targeted for higher end than single core, single interface, single GB > memory systems. > > > > > > Anyway, I think at least ODP_CONFIG_POOLS could be decreased, may be > even halved. Typically, our tests use only one or few pools. Also max > event count could be decreased, but it should be a power of two. It could > be also halved, as most of our tests create only 8k num packets per pool. > > ODP_CONFIG_POOLS used to be 16 and was increased to 64 as a result of > complaints that 16 was too small. The size of the pool being created > is specified as part of the odp_pool_param_t struct passed to > odp_pool_create(). While rounding this up to powers-of-two is > reasonable, there's no need to over-reserve storage that will never be > used. Again, this seems to be an implementation bug. It was never the > intent that the max architectural limits somehow become the minimum > memory requirements needed to support any pool.
This fixed size memory is used for pool global data structure, which includes rings. It's not for the buffer space, that's allocated dynamically. Fixed global structure saves indirections, enables compiler optimizations and is more robust (no need to allocate ring space when pools are created and destroyed). If rings would be separated, it would increase the number of shm allocs and would not save considerable amount of memory (depending on the config values). E.g. with 512k events per pool, each pool needs a 2MB ring and with 32 pools that's 64 MB total, which is not a lot. Also when hugepages are 2MB sized, you would not save memory per pool, as in practice all ring allocations would be rounded up to a 2MB huge page allocation. My point is that, the easiest way forward is to select suitable default values for these configs and consider implementation change only when someone actually has reported a real issue. 272M is large amount but so far it has been only speculated to be an issue. If we cut defaults a bit, those are still large enough (so that nobody notices it) and the memory consumption is small enough (if someone cares). These should be OK for most users ... ODP_CONFIG_POOLS 32 CONFIG_POOL_MAX_NUM (512 * 1024) => 64 MB for rings ... or even a smaller number of events should be more than enough for most users. CONFIG_POOL_MAX_NUM (256 * 1024) => 32 MB for rings -Petri
