[lng-odp] 答复: [PATCH] pool:support to create pool on user's own memory
On 01/14/2016 04:28, Xu Zaibo wrote:
> Hi,
>
> Firstly, I don't want to bring any new type of pool into ODP.
>
> I just want to give user more chance to use more areas of memory. For
> example, referring to your example of various HW offload engines, if there
> two or more types of engines
>
> which have different request on memory attributions exist in the same system,
> which means that one odp_shm_reserve function cannot cover the attributions.
> What should we do?
In that case odp application developer should not care about any
internal memory allocation resources and its. I.e. app has to be portable.
How it will be portable if each app will allocate memory differently?
Just as app need to allocate memory differently, app creates the pool on its
own memory and app uses the pool.
>
> Moreover, ODP's odp_shm_reserve function cannot cover all kinds requests of
> users, even if we use the parameter of flags to define different kind of
> memory in the fucntion of odp_shm_reserve
>
> in my opinion. And user gives its own reserve function, which should be keep
> safe by user. For example, user have to make sure engines can access the
> packets from its own memory.
What is use case?
app does not know about underlying system anything, about memory layout,
numa nodes and etc. Implementation knows about hardware everything. All
of that should be internal (i.e. inside implementation.).
So these things is done by ODP shm as before. This adding of pool param doesn't
change anything before. After this adding, App can uses it own pool on its own
memory.
Regards,
Zaibo Xu.
> -邮件原件-
> 发件人: Savolainen, Petri (Nokia - FI/Espoo) [mailto:petri.savolai...@nokia.com]
> 发送时间: 2016年1月13日 17:49
> 收件人: Xu Zaibo; Tangchaofei; Liuzhongzhu; chenlizhi; yangfajun; Huwei (Xavier)
> 抄送: lng-odp@lists.linaro.org
> 主题: RE: [lng-odp] [PATCH] pool:support to create pool on user's own memory
>
> Hi,
>
> What object types you'd allocate from the pool? Packets, buffers, timeouts or
> some other application specific object?
>
> If it's application specific objects, we'd actually need to define a new pool
> type which can store references to any object type (e.g. uintptr_t). You
> could store pointers to memory blocks, table indexes, etc - basically
> anything.
>
> This modification seems to expect that any implementation can form any type
> of pool from any address the user callback function returns to it.
> Linux-generic could do that (if memory is shared), but it would be
> troublesome to other implementations. For example, various HW offload engines
> (queues, scheduler, crypto, pktin, pktout, tm) must be able to access packets.
>
>
> -Petri
>
>
>
>> -Original Message-
>> From: lng-odp [mailto:lng-odp-boun...@lists.linaro.org] On Behalf Of
>> EXT Zaibo Xu
>> Sent: Tuesday, January 12, 2016 9:23 AM
>> To: tangchao...@huawei.com; liuzhong...@huawei.com;
>> chenli...@huawei.com; yangfa...@huawei.com; xavier.hu...@huawei.com;
>> xuza...@huawei.com
>> Cc: lng-odp@lists.linaro.org
>> Subject: [lng-odp] [PATCH] pool:support to create pool on user's own
>> memory
>>
>> As user initiates pool params with its own memory reserving and un-
>> reserving functions, pool will be created on its own memory, or on odp
>> share memory as before.
>>
>> Signed-off-by: Zaibo Xu
>> ---
>> include/odp/api/pool.h| 12
>> platform/linux-generic/odp_pool.c | 34
>> +-
>> 2 files changed, 37 insertions(+), 9 deletions(-)
>>
>> diff --git a/include/odp/api/pool.h b/include/odp/api/pool.h index
>> 2e79a55..fce0592 100644
>> --- a/include/odp/api/pool.h
>> +++ b/include/odp/api/pool.h
>> @@ -40,6 +40,12 @@ extern "C" {
>> /** Maximum queue name length in chars */ #define ODP_POOL_NAME_LEN
>> 32
>>
>> +/** For user's memory reserve function */ typedef void*
>> +(*odp_usr_resv_mem_t)(uint32_t len);
>> +
>> +/** For user's memory un-reserve function */ typedef int
>> +(*odp_usr_unresv_mem_t)(void *addr);
>> +
>> /**
>>* Pool parameters
>>* Used to communicate pool creation options.
>> @@ -48,6 +54,12 @@ typedef struct odp_pool_param_t {
>> /** Pool type */
>> int type;
>>
>> +/** Odp pool can be created on user's own memory.
>> +there are memory reserve and un-reserve
>> +functions of user. */
>> +odp_usr_resv_mem_t resv_mem_fn;
>> +odp_usr_unresv_mem_t unresv_mem_fn;
>> +
>> union {
>> struct {
>> /** Number of buffers in the pool */ diff --git
>> a/platform/linux-generic/odp_pool.c b/platform/linux-
>> generic/odp_pool.c index 84d35bf..eb8b153 100644
>> --- a/platform/linux-generic/odp_pool.c
>> +++ b/platform/linux-generic/odp_pool.c
>> @@ -295,15 +295,27 @@ odp_pool_t odp_pool_create(const char *name,
>> odp_pool_param_t *params)
>>mdata_size +
>>udata_size);
>>
>>
[lng-odp] 答复: [PATCH] pool:support to create pool on user's own memory
Hi, So, it is sure that implementation may need different kinds of memory. After adding user's own memory operations to the pool param, all the functions of odp_pool_create before are not changed. An implementation is still asking memory from ODP shm or some other internal memory as before. I don’t understand that why implementation will ask memory from application, as ODP has memory pools and implementation should know when and where to create what kind of pool. Finally, implementation can have kinds of memory, user also has kinds of memory and user may intend to create the ODP pool on its own kind of memory to adapt its module to other modules of ODP. User creates its own pool, after using, destroys it. Meanwhile, implementation creates pool, uses pool and destroys it. No matter adding user's own memory operations to the pool param or not, the guy using a pool should know what is the pool using for and what is the pool's attributions. Zaibo Xu -邮件原件- 发件人: Savolainen, Petri (Nokia - FI/Espoo) [mailto:petri.savolai...@nokia.com] 发送时间: 2016年1月14日 23:00 收件人: Xu Zaibo; Tangchaofei; Liuzhongzhu; chenlizhi; yangfajun; Huwei (Xavier) 抄送: lng-odp@lists.linaro.org 主题: RE: [lng-odp] [PATCH] pool:support to create pool on user's own memory Hi, So, it would be memory for any pool type. If an implementation asks memory from the application, portability becomes a problem. How application would know which kind of memory each implementation would need for each pool type? Linux-generic uses ODP shm to reserve memory for pools, but some implementation would prefer to use e.g. SoC internal memory (SRAM, instead of DRAM) for packet pools that are small enough to fit there. This was one of the reasons we removed shm from pool parameters earlier (application don't create and pass memory for pools, but implementation does it by itself). So, the question is how application could tell implementation to prefer memory X for one pool and memory Y for another pool. Are there generic parameters that could describe properties of these two memories. -Petri > -Original Message- > From: EXT Xu Zaibo [mailto:xuza...@huawei.com] > Sent: Thursday, January 14, 2016 3:28 AM > To: Savolainen, Petri (Nokia - FI/Espoo); Tangchaofei; Liuzhongzhu; > chenlizhi; yangfajun; Huwei (Xavier) > Cc: lng-odp@lists.linaro.org > Subject: 答复: [lng-odp] [PATCH] pool:support to create pool on user's > own memory > > Hi, > > Firstly, I don't want to bring any new type of pool into ODP. > > I just want to give user more chance to use more areas of memory. For > example, referring to your example of various HW offload engines, if > there two or more types of engines > > which have different request on memory attributions exist in the same > system, which means that one odp_shm_reserve function cannot cover the > attributions. What should we do? > > Moreover, ODP's odp_shm_reserve function cannot cover all kinds > requests of users, even if we use the parameter of flags to define > different kind of memory in the fucntion of odp_shm_reserve > > in my opinion. And user gives its own reserve function, which should > be keep safe by user. For example, user have to make sure engines can > access the packets from its own memory. > > So let's try to give user chance. Thanks. > > Have a good day > > Zaibu Xu > > > > -邮件原件- > 发件人: Savolainen, Petri (Nokia - FI/Espoo) > [mailto:petri.savolai...@nokia.com] > 发送时间: 2016年1月13日 17:49 > 收件人: Xu Zaibo; Tangchaofei; Liuzhongzhu; chenlizhi; yangfajun; Huwei > (Xavier) > 抄送: lng-odp@lists.linaro.org > 主题: RE: [lng-odp] [PATCH] pool:support to create pool on user's own > memory > > Hi, > > What object types you'd allocate from the pool? Packets, buffers, > timeouts or some other application specific object? > > If it's application specific objects, we'd actually need to define a > new pool type which can store references to any object type (e.g. uintptr_t). > You could store pointers to memory blocks, table indexes, etc - > basically anything. > > This modification seems to expect that any implementation can form any > type of pool from any address the user callback function returns to it. > Linux-generic could do that (if memory is shared), but it would be > troublesome to other implementations. For example, various HW offload > engines (queues, scheduler, crypto, pktin, pktout, tm) must be able to > access packets. > > > -Petri > > > > > -Original Message- > > From: lng-odp [mailto:lng-odp-boun...@lists.linaro.org] On Behalf Of > > EXT Zaibo Xu > > Sent: Tuesday, January 12, 2016 9:23 AM > > To: tangchao...@huawei.com; liuzhong...@huawei.com; > > chenli...@huawei.com; yangfa...@huawei.com; xavier.hu...@huawei.com; > > xuza...@huawei.com > > Cc: lng-odp@lists.linaro.org > > Subject: [lng-odp] [PATCH] pool:support to create pool on user's own > > memory > > > > As user initiates pool params with its own memory reserving an
Re: [lng-odp] 答复: [PATCH] pool:support to create pool on user's own memory
On 01/14/2016 04:28, Xu Zaibo wrote:
Hi,
Firstly, I don't want to bring any new type of pool into ODP.
I just want to give user more chance to use more areas of memory. For example,
referring to your example of various HW offload engines, if there two or more
types of engines
which have different request on memory attributions exist in the same system,
which means that one odp_shm_reserve function cannot cover the attributions.
What should we do?
In that case odp application developer should not care about any
internal memory allocation resources and its. I.e. app has to be portable.
How it will be portable if each app will allocate memory differently?
Moreover, ODP's odp_shm_reserve function cannot cover all kinds requests of
users, even if we use the parameter of flags to define different kind of memory
in the fucntion of odp_shm_reserve
in my opinion. And user gives its own reserve function, which should be keep
safe by user. For example, user have to make sure engines can access the
packets from its own memory.
What is use case?
app does not know about underlying system anything, about memory layout,
numa nodes and etc. Implementation knows about hardware everything. All
of that should be internal (i.e. inside implementation.).
Regards,
Maxim.
So let's try to give user chance. Thanks.
Have a good day
Zaibu Xu
-邮件原件-
发件人: Savolainen, Petri (Nokia - FI/Espoo) [mailto:petri.savolai...@nokia.com]
发送时间: 2016年1月13日 17:49
收件人: Xu Zaibo; Tangchaofei; Liuzhongzhu; chenlizhi; yangfajun; Huwei (Xavier)
抄送: lng-odp@lists.linaro.org
主题: RE: [lng-odp] [PATCH] pool:support to create pool on user's own memory
Hi,
What object types you'd allocate from the pool? Packets, buffers, timeouts or
some other application specific object?
If it's application specific objects, we'd actually need to define a new pool
type which can store references to any object type (e.g. uintptr_t). You could
store pointers to memory blocks, table indexes, etc - basically anything.
This modification seems to expect that any implementation can form any type of
pool from any address the user callback function returns to it. Linux-generic
could do that (if memory is shared), but it would be troublesome to other
implementations. For example, various HW offload engines (queues, scheduler,
crypto, pktin, pktout, tm) must be able to access packets.
-Petri
-Original Message-
From: lng-odp [mailto:lng-odp-boun...@lists.linaro.org] On Behalf Of
EXT Zaibo Xu
Sent: Tuesday, January 12, 2016 9:23 AM
To: tangchao...@huawei.com; liuzhong...@huawei.com;
chenli...@huawei.com; yangfa...@huawei.com; xavier.hu...@huawei.com;
xuza...@huawei.com
Cc: lng-odp@lists.linaro.org
Subject: [lng-odp] [PATCH] pool:support to create pool on user's own
memory
As user initiates pool params with its own memory reserving and un-
reserving functions, pool will be created on its own memory, or on odp
share memory as before.
Signed-off-by: Zaibo Xu
---
include/odp/api/pool.h| 12
platform/linux-generic/odp_pool.c | 34
+-
2 files changed, 37 insertions(+), 9 deletions(-)
diff --git a/include/odp/api/pool.h b/include/odp/api/pool.h index
2e79a55..fce0592 100644
--- a/include/odp/api/pool.h
+++ b/include/odp/api/pool.h
@@ -40,6 +40,12 @@ extern "C" {
/** Maximum queue name length in chars */ #define ODP_POOL_NAME_LEN
32
+/** For user's memory reserve function */ typedef void*
+(*odp_usr_resv_mem_t)(uint32_t len);
+
+/** For user's memory un-reserve function */ typedef int
+(*odp_usr_unresv_mem_t)(void *addr);
+
/**
* Pool parameters
* Used to communicate pool creation options.
@@ -48,6 +54,12 @@ typedef struct odp_pool_param_t {
/** Pool type */
int type;
+ /** Odp pool can be created on user's own memory.
+ there are memory reserve and un-reserve
+ functions of user. */
+ odp_usr_resv_mem_t resv_mem_fn;
+ odp_usr_unresv_mem_t unresv_mem_fn;
+
union {
struct {
/** Number of buffers in the pool */ diff --git
a/platform/linux-generic/odp_pool.c b/platform/linux-
generic/odp_pool.c index 84d35bf..eb8b153 100644
--- a/platform/linux-generic/odp_pool.c
+++ b/platform/linux-generic/odp_pool.c
@@ -295,15 +295,27 @@ odp_pool_t odp_pool_create(const char *name,
odp_pool_param_t *params)
mdata_size +
udata_size);
- shm = odp_shm_reserve(pool->s.name,
- pool->s.pool_size,
- ODP_PAGE_SIZE, 0);
- if (shm == ODP_SHM_INVALID) {
- POOL_UNLOCK(&pool->s.lock);
- return ODP_POOL_INVALID;
+ if (pool->s.params.resv_mem_fn != NULL) {
+ void *addr = pool->s.params.resv_m
[lng-odp] 答复: [PATCH] pool:support to create pool on user's own memory
Hi,
Firstly, I don't want to bring any new type of pool into ODP.
I just want to give user more chance to use more areas of memory. For example,
referring to your example of various HW offload engines, if there two or more
types of engines
which have different request on memory attributions exist in the same system,
which means that one odp_shm_reserve function cannot cover the attributions.
What should we do?
Moreover, ODP's odp_shm_reserve function cannot cover all kinds requests of
users, even if we use the parameter of flags to define different kind of memory
in the fucntion of odp_shm_reserve
in my opinion. And user gives its own reserve function, which should be keep
safe by user. For example, user have to make sure engines can access the
packets from its own memory.
So let's try to give user chance. Thanks.
Have a good day
Zaibu Xu
-邮件原件-
发件人: Savolainen, Petri (Nokia - FI/Espoo) [mailto:petri.savolai...@nokia.com]
发送时间: 2016年1月13日 17:49
收件人: Xu Zaibo; Tangchaofei; Liuzhongzhu; chenlizhi; yangfajun; Huwei (Xavier)
抄送: lng-odp@lists.linaro.org
主题: RE: [lng-odp] [PATCH] pool:support to create pool on user's own memory
Hi,
What object types you'd allocate from the pool? Packets, buffers, timeouts or
some other application specific object?
If it's application specific objects, we'd actually need to define a new pool
type which can store references to any object type (e.g. uintptr_t). You could
store pointers to memory blocks, table indexes, etc - basically anything.
This modification seems to expect that any implementation can form any type of
pool from any address the user callback function returns to it. Linux-generic
could do that (if memory is shared), but it would be troublesome to other
implementations. For example, various HW offload engines (queues, scheduler,
crypto, pktin, pktout, tm) must be able to access packets.
-Petri
> -Original Message-
> From: lng-odp [mailto:lng-odp-boun...@lists.linaro.org] On Behalf Of
> EXT Zaibo Xu
> Sent: Tuesday, January 12, 2016 9:23 AM
> To: tangchao...@huawei.com; liuzhong...@huawei.com;
> chenli...@huawei.com; yangfa...@huawei.com; xavier.hu...@huawei.com;
> xuza...@huawei.com
> Cc: lng-odp@lists.linaro.org
> Subject: [lng-odp] [PATCH] pool:support to create pool on user's own
> memory
>
> As user initiates pool params with its own memory reserving and un-
> reserving functions, pool will be created on its own memory, or on odp
> share memory as before.
>
> Signed-off-by: Zaibo Xu
> ---
> include/odp/api/pool.h| 12
> platform/linux-generic/odp_pool.c | 34
> +-
> 2 files changed, 37 insertions(+), 9 deletions(-)
>
> diff --git a/include/odp/api/pool.h b/include/odp/api/pool.h index
> 2e79a55..fce0592 100644
> --- a/include/odp/api/pool.h
> +++ b/include/odp/api/pool.h
> @@ -40,6 +40,12 @@ extern "C" {
> /** Maximum queue name length in chars */ #define ODP_POOL_NAME_LEN
> 32
>
> +/** For user's memory reserve function */ typedef void*
> +(*odp_usr_resv_mem_t)(uint32_t len);
> +
> +/** For user's memory un-reserve function */ typedef int
> +(*odp_usr_unresv_mem_t)(void *addr);
> +
> /**
> * Pool parameters
> * Used to communicate pool creation options.
> @@ -48,6 +54,12 @@ typedef struct odp_pool_param_t {
> /** Pool type */
> int type;
>
> + /** Odp pool can be created on user's own memory.
> + there are memory reserve and un-reserve
> + functions of user. */
> + odp_usr_resv_mem_t resv_mem_fn;
> + odp_usr_unresv_mem_t unresv_mem_fn;
> +
> union {
> struct {
> /** Number of buffers in the pool */ diff --git
> a/platform/linux-generic/odp_pool.c b/platform/linux-
> generic/odp_pool.c index 84d35bf..eb8b153 100644
> --- a/platform/linux-generic/odp_pool.c
> +++ b/platform/linux-generic/odp_pool.c
> @@ -295,15 +295,27 @@ odp_pool_t odp_pool_create(const char *name,
> odp_pool_param_t *params)
> mdata_size +
> udata_size);
>
> - shm = odp_shm_reserve(pool->s.name,
> - pool->s.pool_size,
> - ODP_PAGE_SIZE, 0);
> - if (shm == ODP_SHM_INVALID) {
> - POOL_UNLOCK(&pool->s.lock);
> - return ODP_POOL_INVALID;
> + if (pool->s.params.resv_mem_fn != NULL) {
> + void *addr = pool->s.params.resv_mem_fn(
> + (uint32_t)pool->s.pool_size);
> +
> + if (addr == NULL) {
> + POOL_UNLOCK(&pool->s.lock);
> + return ODP_POOL_INVALID;
> + }
> + pool->s.pool_base_addr = (uint8_t *)addr;
> + pool->s.pool_shm = (odp_shm_t)!ODP_S
[lng-odp] 答复: [PATCH] pool:support to create pool on user's own memory
Hi Bill,
Thanks for your advices.
1. Sorry for not being marked API-NEXT, as this is my first patch for LNG.
2. Generally, my intention for this mending of pool API is from most of our
product requests. Sometimes, they just want to use the memory of themselves.
Moreover, they even don’t like using of odp_shm_reserve in ODP, as their memory
system may be affected by ODP’s odp_shm_reserve.
3. Seemly, I think this additional API can be NUMA memory or some other
special memory operations from users.
More detailed comments inline
发件人: Bill Fischofer [mailto:bill.fischo...@linaro.org]
发送时间: 2016年1月12日 21:10
收件人: Xu Zaibo
抄送: Tangchaofei; Liuzhongzhu; chenlizhi; yangfajun; Huwei (Xavier); LNG ODP
Mailman List
主题: Re: [lng-odp] [PATCH] pool:support to create pool on user's own memory
Some quick comments:
1. This patch should be marked API-NEXT since it proposes an API change
2. In general I like the idea behind this patch, however I'd also like to see
it incorporate NUMA considerations (perhaps that's why it's being proposed?) as
that's one of the areas that we need to add to ODP pool management. The
approach seems correct to extend the odp_pool_param_t with additional
capabilities. We've discussed adding NUMA zone information to this structure
as well, so perhaps these can be combined.
More detailed comments inline
On Tue, Jan 12, 2016 at 1:22 AM, Zaibo Xu
mailto:xuza...@huawei.com>> wrote:
As user initiates pool params with its own memory reserving and un-reserving
functions,
pool will be created on its own memory, or on odp share memory as before.
Signed-off-by: Zaibo Xu mailto:xuza...@huawei.com>>
---
include/odp/api/pool.h| 12
platform/linux-generic/odp_pool.c | 34 +-
2 files changed, 37 insertions(+), 9 deletions(-)
diff --git a/include/odp/api/pool.h b/include/odp/api/pool.h
index 2e79a55..fce0592 100644
--- a/include/odp/api/pool.h
+++ b/include/odp/api/pool.h
@@ -40,6 +40,12 @@ extern "C" {
/** Maximum queue name length in chars */
#define ODP_POOL_NAME_LEN 32
+/** For user's memory reserve function */
+typedef void* (*odp_usr_resv_mem_t)(uint32_t len);
Why not use the same parameter count/format as odp_shm_reserve() here? That
would seem to offer the most flexibility. Also, for symmetry a better name for
this function might be odp_user_mem_reserve()
I think it’s a good idea of using the same parameters as odp_shm_reserve with
odp_user_mem_reserve() instead.
+
+/** For user's memory un-reserve function */
+typedef int (*odp_usr_unresv_mem_t)(void *addr);
For symmetry with odp_shm_free() a better name here might be odp_user_mem_free()
Also for odp_user_mem_free() instead.
+
/**
* Pool parameters
* Used to communicate pool creation options.
@@ -48,6 +54,12 @@ typedef struct odp_pool_param_t {
/** Pool type */
int type;
+ /** Odp pool can be created on user's own memory.
+ there are memory reserve and un-reserve
+ functions of user. */
+ odp_usr_resv_mem_t resv_mem_fn;
+ odp_usr_unresv_mem_t unresv_mem_fn;
+
union {
struct {
/** Number of buffers in the pool */
diff --git a/platform/linux-generic/odp_pool.c
b/platform/linux-generic/odp_pool.c
index 84d35bf..eb8b153 100644
--- a/platform/linux-generic/odp_pool.c
+++ b/platform/linux-generic/odp_pool.c
@@ -295,15 +295,27 @@ odp_pool_t odp_pool_create(const char *name,
odp_pool_param_t *params)
mdata_size +
udata_size);
- shm = odp_shm_reserve(pool->s.name,
- pool->s.pool_size,
- ODP_PAGE_SIZE, 0);
- if (shm == ODP_SHM_INVALID) {
- POOL_UNLOCK(&pool->s.lock);
- return ODP_POOL_INVALID;
+ if (pool->s.params.resv_mem_fn != NULL) {
+ void *addr = pool->s.params.resv_mem_fn(
+ (uint32_t)pool->s.pool_size);
+
+ if (addr == NULL) {
+ POOL_UNLOCK(&pool->s.lock);
+ return ODP_POOL_INVALID;
+ }
+ pool->s.pool_base_addr = (uint8_t *)addr;
+ pool->s.pool_shm = (odp_shm_t)!ODP_SHM_INVALID;
This is used as an indication that the pool entry is unallocated. Need to
modify the odp_pool_create() routine to account for this change. Similarly,
odp_pool_destroy() needs modification to account for this as does
odp_pool_term_global()
Right now， I still cannot figure out how to initiate ‘pool->s.pool_shm’, so
just using this awkward way. Hopefully , we can have a special macro as
ODP_SHM_INVALID to indicate that this pool memory is allocated from user’s own
memory.
+
