Re: [lng-odp] [API-NEXTv5 1/6] linux-gen: _ishm: adding buddy and slab allocation

[Christophe Milard]

On 12 January 2017 at 08:52, Yi He  wrote:
> Hi, Christophe,
>
> Q1: this patchset name seems a little problem: API-NEXTv5 -> API-NEXT
> PATCHv5

oh god!  Will fix in V6!

>
> Q2: the 1/6 patch cannot be applied to the api-next, I've tried to fix to
> continue the test but did not work, need to re-base and I'll test then.

OK. will rebase for v6

>
> For the patchset code looks OK to me and below questions are to clarify the
> usage of these APIs:
>
> Q3: In the practical usage does it mean all ishmpool creations need to be
> happened in
> 1st control thread before all other threads spawning?

no: you can do that, but it is not needed:
If a pool is created during ODP initialisation (typicaly during one
the xxx_init_global functions), then a normal _ishm_reserve(), with
flags=0 can be performed: as you said, the memory allocated will be
inherited by all.
My first intention was to create a pool this way for the driver
interface so that all drivers could share the same pool. I am coming
from an embedded system world where memory consumption matters :-)
Petri objected that this would make memory corruption harder to debug,
which is true: if a driver exceeds its buffer size, that could make
another driver crash. If each driver create its own pools, this
becomes much less probable.
But this is not a problem either (except that this meant that I had to
expose the pool creation/deletion to the drv API), because _ishm
support the _ODP_ISHM_SINGLE_VA flag, which makes any memory allocated
at any time visible to all ODP threads (inclusive linux thread). So
your question really relates to _ishm and not _ishmpool. I suggest you
read the large comment I made at the top of
platform/linux-generic/_ishm.c which describes that.
>
> Otherwise if a worker ODP thread spawn firstly and after then create an
> ishmpool, the
> memories allocated from this pool seems won't be accessible to other ODP
> threads? because
> the shm_attach and mmap did not happen in other ODP threads (processes).

no: flag  _ODP_ISHM_SINGLE_VA can workaround this issue.

>
> So control thread needs to collect all worker threads requirements on small
> memory size
> requirements and creates all ishmpools in a batch manner before spawning
> worker threads?

no. same as above.

>
> Understand the typical usage:
>
> If the size requirements of small memory are not very specific, create one
> or several pools of
> range [min, max] to support small memory allocation.
>
> If the size requirements of small memory are specific, for example to
> specific data
> structure or small packets, create several pools of dedicated [size].
>
> Thanks and best regards, Yi

Yi: You really comment on the foundamentals, and I REALLY appreciate
that! Your comment are trully thought! What you missed here was the
_ishm capability to allocate memory after fork and still guarantee
sharability of addresses among all ODPthread (linux threads or
processes).
I suggest you read that comment in platform/linux-generic/_ishm.c.

Within the ODP code one can use the _odp_ishm_pool_create(...flags...) function:
Note the flags argument there: this flags urgument is passed directely
to _ishm_reserve().
Typically, during any XXX_init_global functions, flag would be 0.
Because you are running from the OPD instantiation, you are guaranteed
that any memory allocated at this point will be inherited by all.
If ODP later needs a fully sharable pool, it would propably call
_odp_ishm_pool_create() with flag set to _ODP_ISHM_SINGLE_VA set,
hence requiring fully sharable memory.

One could use this flag at all times, but this would consume of the
pre-reserved address space for no reason when run from init.

The function exposed to the driver, odpdrv_shm_pool_create(), always set
this flag, so driver can create fully sharable pools. At any time.

I will fix the rebase and a "PATCH" in the title and send a v6.

Thanks for you review,

Christophe.
>
> On 3 January 2017 at 23:10, Christophe Milard 
> wrote:
>>
>> _ishm now provides functions to create/destroy pools for buddy/slab
>> memory allocation, as well as functions to allocated/release memory
>>
>> from the created pools.
>>
>> Signed-off-by: Christophe Milard 
>> ---
>>  platform/linux-generic/Makefile.am |   2 +
>>  platform/linux-generic/_ishm.c |  14 +-
>>  platform/linux-generic/_ishmpool.c | 811
>> +
>>  .../linux-generic/include/_ishmpool_internal.h |  56 ++
>>  4 files changed, 882 insertions(+), 1 deletion(-)
>>  create mode 100644 platform/linux-generic/_ishmpool.c
>>  create mode 100644 platform/linux-generic/include/_ishmpool_internal.h
>>
>> diff --git a/platform/linux-generic/Makefile.am
>> b/platform/linux-generic/Makefile.am
>> index 999a7f5..d153c5d 100644
>> --- a/platform/linux-generic/Makefile.am
>> +++ b/platform/linux-generic/Makefile.am
>> @@ -127,6 +127,7 @@ noinst_HEADERS = \
>>   ${srcdir}/include/_fdserver_internal.h \
>>  

[lng-odp] master release status

[Savolainen, Petri (Nokia - FI/Espoo)]

Hi,

What's the status of upgrading master with latest api-next commits and tagging 
it? I can see that next branch is now at ...

commit 23e7745272bd405483da737824af25e2e18c8b21
Author: Bill Fischofer 
Date:   Tue Jan 10 09:59:40 2017 -0600

linux-generic: pool: defer ring allocation until pool creation


... which was considered to be needed for release. I think we should merge also 
the next 5 commits from api-next  since those are not API changes. So, this 
should be the tip of next / new release.


commit da32d187cee59c0cb0c5ddf4fb0b22d387e8d951
Author: Petri Savolainen 
Date:   Tue Jan 10 11:19:09 2017 +0200

validation: packet: limit number of failed asserts


Packet references can be left for the next release. So, the diff between master 
and api-next would be only the  addition of new ipsec and packet ref APIs 
(everything else would be the same).


-Petri

[lng-odp] API next SIGABRT on classification test?

[Christophe Milard]

Am I alone seeing that?

PASS: validation/api/pktio/pktio_run.sh
SKIP: validation/api/pktio/pktio_run_tap.sh
PASS: validation/api/shmem/shmem_linux
PASS: ../../test/common_plat/validation/api/atomic/atomic_main
PASS: ../../test/common_plat/validation/api/barrier/barrier_main
PASS: ../../test/common_plat/validation/api/buffer/buffer_main
../../test-driver: line 107: 17971 Aborted (core
dumped) "$@" > $log_file 2>&1
FAIL: ../../test/common_plat/validation/api/classification/classification_main
PASS: ../../test/common_plat/validation/api/cpumask/cpuma

Gdb:
Suite: classification pmr tests
  Test: classification_test_pmr_term_tcp_dport
...odp_packet_io.c:216:setup_pktio_entry():loop uses loop
_ishm.c:448:create_file():open failed for
/dev/hugepages/odp-17500-ishm-pool_ring_1: Permission denied.
_fdserver.c:277:_odp_fdserver_register_fd():FD client register:
pid=17500 key=10, fd=13
_fdserver.c:461:handle_request():storing {ctx=1, key=10}->fd=15
_ishm.c:448:create_file():open failed for
/dev/hugepages/odp-17500-ishm-DefaultPool: Permission denied.
_fdserver.c:277:_odp_fdserver_register_fd():FD client register:
pid=17500 key=11, fd=14
_fdserver.c:461:handle_request():storing {ctx=1, key=11}->fd=16
_ishm.c:448:create_file():open failed for
/dev/hugepages/odp-17500-ishm-pool_ring_2: Permission denied.
_fdserver.c:277:_odp_fdserver_register_fd():FD client register:
pid=17500 key=12, fd=15
_fdserver.c:461:handle_request():storing {ctx=1, key=12}->fd=17
_ishm.c:448:create_file():open failed for
/dev/hugepages/odp-17500-ishm-tcp_dport1: Permission denied.
_fdserver.c:277:_odp_fdserver_register_fd():FD client register:
pid=17500 key=13, fd=16
_fdserver.c:461:handle_request():storing {ctx=1, key=13}->fd=18
./include/odp_packet_internal.h:327:packet_set_len():packet_ref_count(pkt_hdr)
== 1

Program received signal SIGABRT, Aborted.
0x76d89428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/l

Christophe

[lng-odp] [API-NEXT PATCHv6 0/6] small memory amount allocator for drv shm

[Christophe Milard]

Since V5:
 -title fix (Yi)
 -rebased
 -flag name fix (Christophe)

Since V4:
 -fix for clang

Since V3:
 -copyright year changed

Since V2:
 -comment and doxygen fixes (Petri)

Since V1:
 -Common south pool (for all drivers) suppressed. (Petri)
 -function to create and destroy the pools exposed to the interface,
  so that each user create his own pools (Petri)
 -support for fixed size allocator added (Petri, Bill)
 -Creation parameter now includes the min-alloc/max_alloc (Petri)
 -Pool creation parameter passed as struct (Petri)

This patch series aims at giving the ability for units using the ODP
drv (south) interface to allocate small unit of memory (hence going lower
than the page size, which is the lower limit for odp_drvshm_reserve()).
South interface users can create and destroy pool of memory using
odpdrv_shm_pool_create() odpdrv_shm_pool_destroy().
The interface also provides two functions: odpdrv_shm_pool_alloc()
and odpdrv_shm_pool_free() whose usage is very similar to
malloc/free, but which also guarantee address uniqueness amoung all
ODP threads).
Internally, for the linux-gen implementation, this is implemented thanks
to a buddy and slab allocator handling the preallocated memory.
Enumerators typically will need this to contruct the list of enumerated
devices. ODP will also need it for building list of things such as
enumerators, devio, drivers...
(PS:I guess this concept will be needed as well on the north (API) interface
when we start looking into having things working "in process mode")

Christophe Milard (6):
  linux-gen: _ishm: adding buddy and slab allocation
  drv: adding odpdrv_shm_pool functions
  linux-gen: drv: shm: adding pool allocator
  test: drv: shm: adding basic buddy allocation tests
  test: drv: shm: adding basic fixed size allocation tests
  test: drv: shm: adding buddy allocation stress tests

 include/odp/drv/spec/shm.h |  94 +++
 platform/linux-generic/Makefile.am |   2 +
 platform/linux-generic/_ishm.c |  14 +-
 platform/linux-generic/_ishmpool.c | 811 +
 platform/linux-generic/drv_shm.c   |  44 ++
 .../linux-generic/include/_ishmpool_internal.h |  56 ++
 .../linux-generic/include/odp/drv/plat/shm_types.h |   3 +
 .../common_plat/validation/drv/drvshmem/drvshmem.c | 355 +
 .../common_plat/validation/drv/drvshmem/drvshmem.h |   3 +
 9 files changed, 1381 insertions(+), 1 deletion(-)
 create mode 100644 platform/linux-generic/_ishmpool.c
 create mode 100644 platform/linux-generic/include/_ishmpool_internal.h

-- 
2.7.4

[lng-odp] [API-NEXT PATCHv6 2/6] drv: adding odpdrv_shm_pool functions

[Christophe Milard]

Adding functions to create and destroy memory pools (from which memory
can be allocated and freed) are added.
These functions enable the usage of small memory amount (compared to
drvshm_reserve() whose granularity is the page size).
The usage of this pool guatantees that allocated memory is sharable
between ODP threads. (using malloc would not work when ODP threads
are linux processes).

Signed-off-by: Christophe Milard 
Reviewed-by: Petri Savolainen 
---
 include/odp/drv/spec/shm.h | 94 ++
 .../linux-generic/include/odp/drv/plat/shm_types.h |  3 +
 2 files changed, 97 insertions(+)

diff --git a/include/odp/drv/spec/shm.h b/include/odp/drv/spec/shm.h
index ef64f5d..64124c0 100644
--- a/include/odp/drv/spec/shm.h
+++ b/include/odp/drv/spec/shm.h
@@ -220,6 +220,100 @@ int odpdrv_shm_print_all(const char *title);
 uint64_t odpdrv_shm_to_u64(odpdrv_shm_t hdl);
 
 /**
+ * drv shm pool parameters
+ * Used to communicate pool creation options.
+ */
+typedef struct {
+   /** Sum of all (simultaneous) allocs (bytes)*/
+   uint64_t pool_size;
+
+   /** Minimum alloc size user will request from pool (bytes)*/
+   uint64_t min_alloc;
+
+   /** Maximum alloc size user will request from pool (bytes)*/
+   uint64_t max_alloc;
+} odpdrv_shm_pool_param_t;
+
+/**
+ * @typedef odpdrv_shm_pool_t
+ * odpdrv shared memory pool
+ */
+
+/**
+ * Create a memory pool
+ *
+ * This routine is used to create a memory pool. The use of pool name is
+ * optional.
+ * Unique names are not required. However, odpdrv_shm_pool_lookup()
+ * returns only a single matching pool.
+ *
+ * @param name Name of the pool or NULL.
+ * @param paramPool parameters.
+ *
+ * @return Handle of the created drv shm memory pool
+ * @retval ODPDRV_SHM_POOL_INVALID  Pool could not be created
+ */
+odpdrv_shm_pool_t odpdrv_shm_pool_create(const char *pool_name,
+odpdrv_shm_pool_param_t *param);
+
+/**
+ * Destroy a pool previously created by odpdrv_shm_pool_create()
+ *
+ * @param poolHandle of the pool to be destroyed
+ *
+ * @retval 0 Success
+ * @retval <0 Failure
+ *
+ * @note This routine destroys a previously created pool, and will destroy any
+ * internal shared memory objects associated with the pool. Results are
+ * undefined if an attempt is made to destroy a pool that contains allocated
+ * or otherwise active allocations.
+ */
+int odpdrv_shm_pool_destroy(odpdrv_shm_pool_t pool);
+
+/**
+ * Find a memory pool by name
+ *
+ * @param name  Name of the pool
+ *
+ * @return Handle of the first matching pool
+ * @retval ODPDRV_SHM_POOL_INVALID Pool could not be found
+ */
+odpdrv_shm_pool_t odpdrv_shm_pool_lookup(const char *name);
+
+/**
+ * Allocate memory from a memory pool
+ *
+ * @param pool  Memory pool handle
+ * @param size  Number of bytes to allocate (bytes)
+ *
+ * @return A pointer to the allocated memory
+ * @retval NULL on error.
+ */
+void *odpdrv_shm_pool_alloc(odpdrv_shm_pool_t pool, uint64_t size);
+
+/**
+ * Free memory  back to a memory pool
+ *
+ * @param pool  Memory pool handle
+ * @param addr  pointer to a previously allocated memory
+ * (as returned by a previous call to odpdrv_shm_pool_alloc)
+ */
+void odpdrv_shm_pool_free(odpdrv_shm_pool_t pool, void *addr);
+
+/**
+ * Print memory pool info
+ *
+ * @param title A string to be printed as a title (e.g. location)
+ * @param pool  Memory pool handle
+ *
+ * @return 0 on success, negative value if pool inconsistency is detected.
+ *
+ * @note This routine writes implementation-defined information about the
+ * specified pool to the ODP log. The intended use is for debugging.
+ */
+int  odpdrv_shm_pool_print(const char *title, odpdrv_shm_pool_t pool);
+/**
  * @}
  */
 
diff --git a/platform/linux-generic/include/odp/drv/plat/shm_types.h 
b/platform/linux-generic/include/odp/drv/plat/shm_types.h
index c48eeca..50a0837 100644
--- a/platform/linux-generic/include/odp/drv/plat/shm_types.h
+++ b/platform/linux-generic/include/odp/drv/plat/shm_types.h
@@ -35,6 +35,9 @@ static inline uint64_t odpdrv_shm_to_u64(odpdrv_shm_t hdl)
return _odpdrv_pri(hdl);
 }
 
+typedef ODPDRV_HANDLE_T(odpdrv_shm_pool_t);
+
+#define ODPDRV_SHM_POOL_INVALID _odpdrv_cast_scalar(odpdrv_shm_pool_t, NULL)
 /**
  * @}
  */
-- 
2.7.4

[lng-odp] [API-NEXT PATCHv6 3/6] linux-gen: drv: shm: adding pool allocator

[Christophe Milard]

Adding functions to create memory pools and allocate / free memory from
the created pools.
These functions calls their _ishm conterpart, of course.

Signed-off-by: Christophe Milard 
---
 platform/linux-generic/drv_shm.c | 44 
 1 file changed, 44 insertions(+)

diff --git a/platform/linux-generic/drv_shm.c b/platform/linux-generic/drv_shm.c
index 9b2560d..325632e 100644
--- a/platform/linux-generic/drv_shm.c
+++ b/platform/linux-generic/drv_shm.c
@@ -8,6 +8,7 @@
 #include 
 #include 
 #include <_ishm_internal.h>
+#include <_ishmpool_internal.h>
 
 static inline uint32_t from_handle(odpdrv_shm_t shm)
 {
@@ -100,3 +101,46 @@ int odpdrv_shm_print_all(const char *title)
 {
return _odp_ishm_status(title);
 }
+
+odpdrv_shm_pool_t odpdrv_shm_pool_create(const char *pool_name,
+odpdrv_shm_pool_param_t *param)
+{
+   int flags;
+
+   /* force unique address for all ODP threads */
+   flags = _ODP_ISHM_SINGLE_VA;
+   return (odpdrv_shm_pool_t)_odp_ishm_pool_create(pool_name,
+   param->pool_size,
+   param->min_alloc,
+   param->max_alloc,
+   flags);
+}
+
+int odpdrv_shm_pool_destroy(odpdrv_shm_pool_t pool)
+{
+   return _odp_ishm_pool_destroy((_odp_ishm_pool_t *)(void*)pool);
+}
+
+odpdrv_shm_pool_t odpdrv_shm_pool_lookup(const char *name)
+{
+   return (odpdrv_shm_pool_t)_odp_ishm_pool_lookup(name);
+}
+
+void *odpdrv_shm_pool_alloc(odpdrv_shm_pool_t pool, uint64_t size)
+{
+   return _odp_ishm_pool_alloc((_odp_ishm_pool_t *)(void*)pool, size);
+}
+
+void odpdrv_shm_pool_free(odpdrv_shm_pool_t pool, void *addr)
+{
+   (void)_odp_ishm_pool_free((_odp_ishm_pool_t *)(void*)pool, addr);
+}
+
+int odpdrv_shm_pool_print(const char *title, odpdrv_shm_pool_t pool)
+{
+   return _odp_ishm_pool_status(title, (_odp_ishm_pool_t *)(void*)pool);
+}
+
+/**
+ * @}
+ */
-- 
2.7.4

[lng-odp] [API-NEXT PATCHv6 1/6] linux-gen: _ishm: adding buddy and slab allocation

[Christophe Milard]

_ishm now provides functions to create/destroy pools for buddy/slab
memory allocation, as well as functions to allocated/release memory
from the created pools.

Signed-off-by: Christophe Milard 
---
 platform/linux-generic/Makefile.am |   2 +
 platform/linux-generic/_ishm.c |  14 +-
 platform/linux-generic/_ishmpool.c | 811 +
 .../linux-generic/include/_ishmpool_internal.h |  56 ++
 4 files changed, 882 insertions(+), 1 deletion(-)
 create mode 100644 platform/linux-generic/_ishmpool.c
 create mode 100644 platform/linux-generic/include/_ishmpool_internal.h

diff --git a/platform/linux-generic/Makefile.am 
b/platform/linux-generic/Makefile.am
index 999a7f5..d153c5d 100644
--- a/platform/linux-generic/Makefile.am
+++ b/platform/linux-generic/Makefile.am
@@ -127,6 +127,7 @@ noinst_HEADERS = \
  ${srcdir}/include/_fdserver_internal.h \
  ${srcdir}/include/_ishm_internal.h \
  ${srcdir}/include/_ishmphy_internal.h \
+ ${srcdir}/include/_ishmpool_internal.h \
  ${srcdir}/include/odp_align_internal.h \
  ${srcdir}/include/odp_atomic_internal.h \
  ${srcdir}/include/odp_buffer_inlines.h \
@@ -171,6 +172,7 @@ __LIB__libodp_linux_la_SOURCES = \
   _fdserver.c \
   _ishm.c \
   _ishmphy.c \
+  _ishmpool.c \
   odp_atomic.c \
   odp_barrier.c \
   odp_buffer.c \
diff --git a/platform/linux-generic/_ishm.c b/platform/linux-generic/_ishm.c
index 23f620d..4c2578b 100644
--- a/platform/linux-generic/_ishm.c
+++ b/platform/linux-generic/_ishm.c
@@ -59,6 +59,7 @@
 #include <_fdserver_internal.h>
 #include <_ishm_internal.h>
 #include <_ishmphy_internal.h>
+#include <_ishmpool_internal.h>
 #include 
 #include 
 #include 
@@ -1441,8 +1442,19 @@ int _odp_ishm_init_global(void)
 * is performed for the main thread... Many init_global() functions
 * indeed assume the availability of odp_shm_reserve()...:
 */
-   return do_odp_ishm_init_local();
+   if (do_odp_ishm_init_local()) {
+   ODP_ERR("unable to init the main thread\n.");
+   goto init_glob_err4;
+   }
+
+   /* get ready to create pools: */
+   _odp_ishm_pool_init();
 
+   return 0;
+
+init_glob_err4:
+   if (_odp_ishmphy_unbook_va())
+   ODP_ERR("unable to unbook virtual space\n.");
 init_glob_err3:
if (munmap(ishm_ftbl, sizeof(ishm_ftable_t)) < 0)
ODP_ERR("unable to munmap main fragment table\n.");
diff --git a/platform/linux-generic/_ishmpool.c 
b/platform/linux-generic/_ishmpool.c
new file mode 100644
index 000..df6e49e
--- /dev/null
+++ b/platform/linux-generic/_ishmpool.c
@@ -0,0 +1,811 @@
+/* Copyright (c) 2017, Linaro Limited
+ * All rights reserved.
+ *
+ * SPDX-License-Identifier: BSD-3-Clause
+ */
+
+/* This file gathers the buddy and slab allocation functionality provided
+ * by _ishm.
+ * _odp_ishmpool_create() can be used to create a pool for buddy/slab
+ * allocation. _odp_ishmpool_create() will allocate a memory area using
+ * ishm_reserve() for both the control part (needed for tracking
+ * allocation/free...) and the user memory itself (part of which will be given
+ * at each ishmpool_alloc()).
+ * The element size provided at pool creation time determines whether
+ * to pool will of type buddy or slab.
+ * For buddy, all allocations are rounded to the nearest power of 2.
+ *
+ * The implementation of the buddy allocator is very traditional: it
+ * maintains N lists of free buffers.
+ * The control part actually contains these N queue heads, (N-M are actually
+ * used), the free buffers themselves being used for chaining (the chaining 
info
+ * is in the buffers: as they are "free" they should not be touched by the
+ * user). The control part also contains a array of bytes for remembering
+ * the size (actually the order) of the allocated buffers:
+ * There are 2^(N-M) such bytes, this number being the maximum number of
+ * allocated buffers (when all allocation are <= 2^M bytes)
+ * Buddy allocators handle fragmentation by splitting or merging blocks by 2.
+ * They guarantee a minimum efficiency of 50%, at worse case fragmentation.
+ *
+ * Slab implementation is even simpler, all free elements being queued in
+ * one single queue at init, taken from this queue when allocated and
+ * returned to this same queue when freed.
+ *
+ * The reason for not using malloc() is that malloc does not guarantee
+ * memory sharability between ODP threads (regardless of their implememtation)
+ * which ishm_reserve() can do. see the comments around
+ * _odp_ishmbud_pool_create() and ishm_reserve() for more details.
+ *
+ * This file is divided in 3 sections: the first one regroups functions
+ * needed by 

[lng-odp] [API-NEXT PATCHv6 4/6] test: drv: shm: adding basic buddy allocation tests

[Christophe Milard]

Basic tests for odpdrv_shm_pool are added here, creating a buddy
pool and performing basic alloc/free on it

Signed-off-by: Christophe Milard 
---
 .../common_plat/validation/drv/drvshmem/drvshmem.c | 92 ++
 .../common_plat/validation/drv/drvshmem/drvshmem.h |  1 +
 2 files changed, 93 insertions(+)

diff --git a/test/common_plat/validation/drv/drvshmem/drvshmem.c 
b/test/common_plat/validation/drv/drvshmem/drvshmem.c
index 0247a03..5843573 100644
--- a/test/common_plat/validation/drv/drvshmem/drvshmem.c
+++ b/test/common_plat/validation/drv/drvshmem/drvshmem.c
@@ -21,6 +21,12 @@
 #define STRESS_RANDOM_SZ 5
 #define STRESS_ITERATION 5000
 
+#define POOL_NAME "test_pool"
+#define POOL_SZ (1UL << 20)/* 1 MBytes */
+#define TEST_SZ 1000
+#define SZ_1K   1024
+#define BUFF_PATTERN 0xA3
+
 typedef enum {
STRESS_FREE, /* entry is free and can be allocated */
STRESS_BUSY, /* entry is being processed: don't touch */
@@ -762,11 +768,97 @@ void drvshmem_test_stress(void)
CU_ASSERT(odpdrv_shm_print_all("After stress tests") == base);
 }
 
+void drvshmem_test_buddy_basic(void)
+{
+   odpdrv_shm_pool_param_t pool_params;
+   odpdrv_shm_pool_t pool, found_pool;
+   uint8_t *buff;
+   uint8_t *addrs[TEST_SZ];
+   uint8_t length;
+   int i, j;
+
+   /* create a pool and check that it can be looked up */
+   pool_params.pool_size = POOL_SZ;
+   pool_params.min_alloc = 1;
+   pool_params.max_alloc = POOL_SZ;
+   pool = odpdrv_shm_pool_create(POOL_NAME, &pool_params);
+   found_pool = odpdrv_shm_pool_lookup(POOL_NAME);
+   CU_ASSERT(found_pool == pool);
+
+   /* alloc a 1k buffer, filling its contents: */
+   buff = odpdrv_shm_pool_alloc(pool, SZ_1K);
+   CU_ASSERT_PTR_NOT_NULL(buff);
+   for (i = 0; i < SZ_1K; i++)
+   buff[i] = BUFF_PATTERN;
+   odpdrv_shm_pool_print("buddy test: 1K reserved", pool);
+
+   /* alloc as many buffer a possible on increseasing sz */
+   for (i = 0; i < TEST_SZ; i++) {
+   length = i * 16;
+   addrs[i] = odpdrv_shm_pool_alloc(pool, length);
+   /* if alloc was success, fill buffer for later check */
+   if (addrs[i]) {
+   for (j = 0; j < length; j++)
+   addrs[i][j] = (uint8_t)(length & 0xFF);
+   }
+   }
+   odpdrv_shm_pool_print("buddy test: after many mallocs", pool);
+
+   /* release every 3rth buffer, checking contents: */
+   for (i = 0; i < TEST_SZ; i += 3) {
+   /* if buffer was allocated, check the pattern in it */
+   if (addrs[i]) {
+   length = i * 16;
+   for (j = 0; j < length; j++)
+   CU_ASSERT(addrs[i][j] ==
+ (uint8_t)(length & 0xFF));
+   }
+   odpdrv_shm_pool_free(pool, addrs[i]);
+   }
+   odpdrv_shm_pool_print("buddy test: after 1/3 free:", pool);
+
+   /* realloc them:*/
+   for (i = 0; i < TEST_SZ; i += 3) {
+   length = i * 16;
+   addrs[i] = odpdrv_shm_pool_alloc(pool, length);
+   /* if alloc was success, fill buffer for later check */
+   if (addrs[i]) {
+   for (j = 0; j < length; j++)
+   addrs[i][j] = (uint8_t)(length & 0xFF);
+   }
+   }
+   odpdrv_shm_pool_print("buddy test: after realloc:", pool);
+
+   /* free all (except buff), checking contents: */
+   for (i = 0; i < TEST_SZ; i++) {
+   /* if buffer was allocated, check the pattern in it */
+   if (addrs[i]) {
+   length = i * 16;
+   for (j = 0; j < length; j++)
+   CU_ASSERT(addrs[i][j] ==
+ (uint8_t)(length & 0xFF))
+   }
+   odpdrv_shm_pool_free(pool, addrs[i]);
+   }
+   odpdrv_shm_pool_print("buddy test: after all but 1K free:", pool);
+
+   /* check contents of our initial 1K buffer: */
+   for (i = 0; i < SZ_1K; i++)
+   CU_ASSERT((buff[i] == BUFF_PATTERN))
+   odpdrv_shm_pool_free(pool, buff);
+
+   odpdrv_shm_pool_print("buddy test: after all free", pool);
+
+   /* destroy pool: */
+   odpdrv_shm_pool_destroy(pool);
+}
+
 odp_testinfo_t drvshmem_suite[] = {
ODP_TEST_INFO(drvshmem_test_basic),
ODP_TEST_INFO(drvshmem_test_reserve_after_fork),
ODP_TEST_INFO(drvshmem_test_singleva_after_fork),
ODP_TEST_INFO(drvshmem_test_stress),
+   ODP_TEST_INFO(drvshmem_test_buddy_basic),
ODP_TEST_INFO_NULL,
 };
 
diff --git a/test/common_plat/validation/drv/drvshmem/drvshmem.h 
b/test/common_plat/validation/drv/drvshmem/drvshmem.h
index f4c26a1..ab45f7c 100644
--- a/test/common_plat/validation/drv/drvshmem/drvshmem.h
+++ b/t

[lng-odp] [API-NEXT PATCHv6 5/6] test: drv: shm: adding basic fixed size allocation tests

[Christophe Milard]

Basic tests for odpdrv_shm_pool are added here, creating a fixed size
pool and performing basic alloc/free on it

Signed-off-by: Christophe Milard 
---
 .../common_plat/validation/drv/drvshmem/drvshmem.c | 86 ++
 .../common_plat/validation/drv/drvshmem/drvshmem.h |  1 +
 2 files changed, 87 insertions(+)

diff --git a/test/common_plat/validation/drv/drvshmem/drvshmem.c 
b/test/common_plat/validation/drv/drvshmem/drvshmem.c
index 5843573..d4dedea 100644
--- a/test/common_plat/validation/drv/drvshmem/drvshmem.c
+++ b/test/common_plat/validation/drv/drvshmem/drvshmem.c
@@ -853,12 +853,98 @@ void drvshmem_test_buddy_basic(void)
odpdrv_shm_pool_destroy(pool);
 }
 
+void drvshmem_test_slab_basic(void)
+{
+   odpdrv_shm_pool_param_t pool_params;
+   odpdrv_shm_pool_t pool, found_pool;
+   uint8_t *buff;
+   uint8_t *addrs[TEST_SZ];
+   uint16_t length;
+   int i, j;
+
+   /* create a pool and check that it can be looked up */
+   pool_params.pool_size = POOL_SZ;
+   pool_params.min_alloc = SZ_1K; /* constant size will give slab */
+   pool_params.max_alloc = SZ_1K;
+   pool = odpdrv_shm_pool_create(POOL_NAME, &pool_params);
+   found_pool = odpdrv_shm_pool_lookup(POOL_NAME);
+   CU_ASSERT(found_pool == pool);
+
+   /* alloc a 1k buffer, filling its contents: */
+   buff = odpdrv_shm_pool_alloc(pool, SZ_1K);
+   CU_ASSERT_PTR_NOT_NULL(buff);
+   for (i = 0; i < SZ_1K; i++)
+   buff[i] = BUFF_PATTERN;
+   odpdrv_shm_pool_print("buddy test: 1K reserved", pool);
+
+   /* alloc as many 1K buffer a possible */
+   for (i = 0; i < TEST_SZ; i++) {
+   length = SZ_1K;
+   addrs[i] = odpdrv_shm_pool_alloc(pool, length);
+   /* if alloc was success, fill buffer for later check */
+   if (addrs[i]) {
+   for (j = 0; j < length; j++)
+   addrs[i][j] = (uint8_t)(length & 0xFF);
+   }
+   }
+   odpdrv_shm_pool_print("slab test: after many mallocs", pool);
+
+   /* release every 3rth buffer, checking contents: */
+   for (i = 0; i < TEST_SZ; i += 3) {
+   /* if buffer was allocated, check the pattern in it */
+   if (addrs[i]) {
+   length = SZ_1K;
+   for (j = 0; j < length; j++)
+   CU_ASSERT(addrs[i][j] ==
+ (uint8_t)(length & 0xFF));
+   }
+   odpdrv_shm_pool_free(pool, addrs[i]);
+   }
+   odpdrv_shm_pool_print("slab test: after 1/3 free:", pool);
+
+   /* realloc them:*/
+   for (i = 0; i < TEST_SZ; i += 3) {
+   length = SZ_1K;
+   addrs[i] = odpdrv_shm_pool_alloc(pool, length);
+   /* if alloc was success, fill buffer for later check */
+   if (addrs[i]) {
+   for (j = 0; j < length; j++)
+   addrs[i][j] = (uint8_t)(length & 0xFF);
+   }
+   }
+   odpdrv_shm_pool_print("slab test: after realloc:", pool);
+
+   /* free all (except buff), checking contents: */
+   for (i = 0; i < TEST_SZ; i++) {
+   /* if buffer was allocated, check the pattern in it */
+   if (addrs[i]) {
+   length = SZ_1K;
+   for (j = 0; j < length; j++)
+   CU_ASSERT(addrs[i][j] ==
+ (uint8_t)(length & 0xFF))
+   }
+   odpdrv_shm_pool_free(pool, addrs[i]);
+   }
+   odpdrv_shm_pool_print("slab test: after all but 1K free:", pool);
+
+   /* check contents of our initial 1K buffer: */
+   for (i = 0; i < SZ_1K; i++)
+   CU_ASSERT((buff[i] == BUFF_PATTERN))
+   odpdrv_shm_pool_free(pool, buff);
+
+   odpdrv_shm_pool_print("slab test: after all free", pool);
+
+   /* destroy pool: */
+   odpdrv_shm_pool_destroy(pool);
+}
+
 odp_testinfo_t drvshmem_suite[] = {
ODP_TEST_INFO(drvshmem_test_basic),
ODP_TEST_INFO(drvshmem_test_reserve_after_fork),
ODP_TEST_INFO(drvshmem_test_singleva_after_fork),
ODP_TEST_INFO(drvshmem_test_stress),
ODP_TEST_INFO(drvshmem_test_buddy_basic),
+   ODP_TEST_INFO(drvshmem_test_slab_basic),
ODP_TEST_INFO_NULL,
 };
 
diff --git a/test/common_plat/validation/drv/drvshmem/drvshmem.h 
b/test/common_plat/validation/drv/drvshmem/drvshmem.h
index ab45f7c..fdc1080 100644
--- a/test/common_plat/validation/drv/drvshmem/drvshmem.h
+++ b/test/common_plat/validation/drv/drvshmem/drvshmem.h
@@ -15,6 +15,7 @@ void drvshmem_test_reserve_after_fork(void);
 void drvshmem_test_singleva_after_fork(void);
 void drvshmem_test_stress(void);
 void drvshmem_test_buddy_basic(void);
+void drvshmem_test_slab_basic(void);
 
 /* test arrays: */
 extern odp_testinfo_t drvshmem_suite[];

[lng-odp] [API-NEXT PATCHv6 6/6] test: drv: shm: adding buddy allocation stress tests

[Christophe Milard]

Stress tests for the random size allocator (buddy allocator in
linux-generic) are added here.

Signed-off-by: Christophe Milard 
---
 .../common_plat/validation/drv/drvshmem/drvshmem.c | 177 +
 .../common_plat/validation/drv/drvshmem/drvshmem.h |   1 +
 2 files changed, 178 insertions(+)

diff --git a/test/common_plat/validation/drv/drvshmem/drvshmem.c 
b/test/common_plat/validation/drv/drvshmem/drvshmem.c
index d4dedea..0f882ae 100644
--- a/test/common_plat/validation/drv/drvshmem/drvshmem.c
+++ b/test/common_plat/validation/drv/drvshmem/drvshmem.c
@@ -938,6 +938,182 @@ void drvshmem_test_slab_basic(void)
odpdrv_shm_pool_destroy(pool);
 }
 
+/*
+ * thread part for the drvshmem_test_buddy_stress
+ */
+static int run_test_buddy_stress(void *arg ODP_UNUSED)
+{
+   odpdrv_shm_t shm;
+   odpdrv_shm_pool_t pool;
+   uint8_t *address;
+   shared_test_data_t *glob_data;
+   uint8_t random_bytes[STRESS_RANDOM_SZ];
+   uint32_t index;
+   uint32_t size;
+   uint8_t data;
+   uint32_t iter;
+   uint32_t i;
+
+   shm = odpdrv_shm_lookup_by_name(MEM_NAME);
+   glob_data = odpdrv_shm_addr(shm);
+   CU_ASSERT_PTR_NOT_NULL(glob_data);
+
+   /* get the pool to test */
+   pool = odpdrv_shm_pool_lookup(POOL_NAME);
+
+   /* wait for general GO! */
+   odpdrv_barrier_wait(&glob_data->test_barrier1);
+   /*
+
+* at each iteration: pick up a random index for
+* glob_data->stress[index]: If the entry is free, allocated small mem
+* randomly. If it is already allocated, make checks and free it:
+* Note that different tread can allocate or free a given block
+*/
+   for (iter = 0; iter < STRESS_ITERATION; iter++) {
+   /* get 4 random bytes from which index, size ,align, flags
+* and data will be derived:
+*/
+   odp_random_data(random_bytes, STRESS_RANDOM_SZ, 0);
+   index = random_bytes[0] & (STRESS_SIZE - 1);
+
+   odp_spinlock_lock(&glob_data->stress_lock);
+
+   switch (glob_data->stress[index].state) {
+   case STRESS_FREE:
+   /* allocated a new block for this entry */
+
+   glob_data->stress[index].state = STRESS_BUSY;
+   odp_spinlock_unlock(&glob_data->stress_lock);
+
+   size  = (random_bytes[1] + 1) << 4; /* up to 4Kb */
+   data  = random_bytes[2];
+
+   address = odpdrv_shm_pool_alloc(pool, size);
+   glob_data->stress[index].address = address;
+   if (address == NULL) { /* out of mem ? */
+   odp_spinlock_lock(&glob_data->stress_lock);
+   glob_data->stress[index].state = STRESS_ALLOC;
+   odp_spinlock_unlock(&glob_data->stress_lock);
+   continue;
+   }
+
+   glob_data->stress[index].size = size;
+   glob_data->stress[index].data_val = data;
+
+   /* write some data: */
+   for (i = 0; i < size; i++)
+   address[i] = (data++) & 0xFF;
+   odp_spinlock_lock(&glob_data->stress_lock);
+   glob_data->stress[index].state = STRESS_ALLOC;
+   odp_spinlock_unlock(&glob_data->stress_lock);
+
+   break;
+
+   case STRESS_ALLOC:
+   /* free the block for this entry */
+
+   glob_data->stress[index].state = STRESS_BUSY;
+   odp_spinlock_unlock(&glob_data->stress_lock);
+   address = glob_data->stress[index].address;
+
+   if (shm == NULL) { /* out of mem ? */
+   odp_spinlock_lock(&glob_data->stress_lock);
+   glob_data->stress[index].state = STRESS_FREE;
+   odp_spinlock_unlock(&glob_data->stress_lock);
+   continue;
+   }
+
+   /* check that data is reachable and correct: */
+   data = glob_data->stress[index].data_val;
+   size = glob_data->stress[index].size;
+   for (i = 0; i < size; i++) {
+   CU_ASSERT(address[i] == (data & 0xFF));
+   data++;
+   }
+
+   odpdrv_shm_pool_free(pool, address);
+
+   odp_spinlock_lock(&glob_data->stress_lock);
+   glob_data->stress[index].state = STRESS_FREE;
+   odp_spinlock_unlock(&glob_data->stress_lock);
+
+   break;
+
+   case STRESS_BUSY:
+   default:
+   

[lng-odp] [Bug 2812] New: helper/test/process fails on a single core system

[bugzilla-daemon]

https://bugs.linaro.org/show_bug.cgi?id=2812

Bug ID: 2812
   Summary: helper/test/process fails on a single core system
   Product: OpenDataPlane - linux- generic reference
   Version: v1.12.0.0
  Hardware: x86
OS: Linux
Status: UNCONFIRMED
  Severity: normal
  Priority: ---
 Component: General ODP
  Assignee: maxim.uva...@linaro.org
  Reporter: matias@nokia.com
CC: lng-odp@lists.linaro.org
  Target Milestone: ---

Make check fails on a single core VM on 'process' helper test.  This same issue
is also present in the api-next branch.


make check output:

FAIL: process
=

linux.c:140:odph_linux_process_fork_n():Bad num
process.c:74:main():Fork workers failed -1
 PKTIO: initialized loop interface.
 PKTIO: initialized pcap interface.
 PKTIO: initialized socket mmap, use export ODP_PKTIO_DISABLE_SOCKET_MMAP=1 to
disable.
 PKTIO: initialized socket mmsg,use export ODP_PKTIO_DISABLE_SOCKET_MMSG=1 to
disable.
System can only support 1 processes and not the 16 requested
default cpu mask:   0x1
default num worker processes: 1
the first CPU:  0
new cpu mask:   0x0
new num worker processes: 0

-- 
You are receiving this mail because:
You are on the CC list for the bug.

Re: [lng-odp] API next SIGABRT on classification test?

[Christophe Milard]

Seems to be related to the usage of the "--enable-debug" option.
Anyone building with it?

e.g:
./bootstrap && ./configure  --enable-debug  --enable-test-vald   &&
make -j 8 && make check

Christophe.

On 12 January 2017 at 10:20, Christophe Milard
 wrote:
> Am I alone seeing that?
>
> PASS: validation/api/pktio/pktio_run.sh
> SKIP: validation/api/pktio/pktio_run_tap.sh
> PASS: validation/api/shmem/shmem_linux
> PASS: ../../test/common_plat/validation/api/atomic/atomic_main
> PASS: ../../test/common_plat/validation/api/barrier/barrier_main
> PASS: ../../test/common_plat/validation/api/buffer/buffer_main
> ../../test-driver: line 107: 17971 Aborted (core
> dumped) "$@" > $log_file 2>&1
> FAIL: ../../test/common_plat/validation/api/classification/classification_main
> PASS: ../../test/common_plat/validation/api/cpumask/cpuma
>
> Gdb:
> Suite: classification pmr tests
>   Test: classification_test_pmr_term_tcp_dport
> ...odp_packet_io.c:216:setup_pktio_entry():loop uses loop
> _ishm.c:448:create_file():open failed for
> /dev/hugepages/odp-17500-ishm-pool_ring_1: Permission denied.
> _fdserver.c:277:_odp_fdserver_register_fd():FD client register:
> pid=17500 key=10, fd=13
> _fdserver.c:461:handle_request():storing {ctx=1, key=10}->fd=15
> _ishm.c:448:create_file():open failed for
> /dev/hugepages/odp-17500-ishm-DefaultPool: Permission denied.
> _fdserver.c:277:_odp_fdserver_register_fd():FD client register:
> pid=17500 key=11, fd=14
> _fdserver.c:461:handle_request():storing {ctx=1, key=11}->fd=16
> _ishm.c:448:create_file():open failed for
> /dev/hugepages/odp-17500-ishm-pool_ring_2: Permission denied.
> _fdserver.c:277:_odp_fdserver_register_fd():FD client register:
> pid=17500 key=12, fd=15
> _fdserver.c:461:handle_request():storing {ctx=1, key=12}->fd=17
> _ishm.c:448:create_file():open failed for
> /dev/hugepages/odp-17500-ishm-tcp_dport1: Permission denied.
> _fdserver.c:277:_odp_fdserver_register_fd():FD client register:
> pid=17500 key=13, fd=16
> _fdserver.c:461:handle_request():storing {ctx=1, key=13}->fd=18
> ./include/odp_packet_internal.h:327:packet_set_len():packet_ref_count(pkt_hdr)
> == 1
>
> Program received signal SIGABRT, Aborted.
> 0x76d89428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/l
>
> Christophe

Re: [lng-odp] API next SIGABRT on classification test?

[Bill Fischofer]

On Thu, Jan 12, 2017 at 3:20 AM, Christophe Milard
 wrote:
> Am I alone seeing that?

You found a bug that only surfaces if you're running with
--enable-debug. It's a one-line fix that I'll post shortly. Thanks.

>
> PASS: validation/api/pktio/pktio_run.sh
> SKIP: validation/api/pktio/pktio_run_tap.sh
> PASS: validation/api/shmem/shmem_linux
> PASS: ../../test/common_plat/validation/api/atomic/atomic_main
> PASS: ../../test/common_plat/validation/api/barrier/barrier_main
> PASS: ../../test/common_plat/validation/api/buffer/buffer_main
> ../../test-driver: line 107: 17971 Aborted (core
> dumped) "$@" > $log_file 2>&1
> FAIL: ../../test/common_plat/validation/api/classification/classification_main
> PASS: ../../test/common_plat/validation/api/cpumask/cpuma
>
> Gdb:
> Suite: classification pmr tests
>   Test: classification_test_pmr_term_tcp_dport
> ...odp_packet_io.c:216:setup_pktio_entry():loop uses loop
> _ishm.c:448:create_file():open failed for
> /dev/hugepages/odp-17500-ishm-pool_ring_1: Permission denied.
> _fdserver.c:277:_odp_fdserver_register_fd():FD client register:
> pid=17500 key=10, fd=13
> _fdserver.c:461:handle_request():storing {ctx=1, key=10}->fd=15
> _ishm.c:448:create_file():open failed for
> /dev/hugepages/odp-17500-ishm-DefaultPool: Permission denied.
> _fdserver.c:277:_odp_fdserver_register_fd():FD client register:
> pid=17500 key=11, fd=14
> _fdserver.c:461:handle_request():storing {ctx=1, key=11}->fd=16
> _ishm.c:448:create_file():open failed for
> /dev/hugepages/odp-17500-ishm-pool_ring_2: Permission denied.
> _fdserver.c:277:_odp_fdserver_register_fd():FD client register:
> pid=17500 key=12, fd=15
> _fdserver.c:461:handle_request():storing {ctx=1, key=12}->fd=17
> _ishm.c:448:create_file():open failed for
> /dev/hugepages/odp-17500-ishm-tcp_dport1: Permission denied.
> _fdserver.c:277:_odp_fdserver_register_fd():FD client register:
> pid=17500 key=13, fd=16
> _fdserver.c:461:handle_request():storing {ctx=1, key=13}->fd=18
> ./include/odp_packet_internal.h:327:packet_set_len():packet_ref_count(pkt_hdr)
> == 1
>
> Program received signal SIGABRT, Aborted.
> 0x76d89428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/l
>
> Christophe

Re: [lng-odp] [API-NEXT PATCHv6 2/5] linux-generic: packet: implement reference apis

[Savolainen, Petri (Nokia - FI/Espoo)]

This patch is now merged, although I had some doubts that it has bad impact on 
performance. Here are some performance results for couple of simple, single 
thread packet alloc/free test cases. No references involved, just plain packets 
as before.

Test results before and after "linux-generic: packet: implement reference apis":

CPU cycles per operation
before  after
packet_alloc127.8   250.9   +96 %
packet_alloc_multi  873.7   1538.8  +76 %
packet_free 31  116.8   +277 %
packet_free_multi   214.5   1369.2  +538 %
packet_alloc_free   73.4193.7   +164 %
packet_alloc_free_multi 286.1   1228.8  +330 % 


Huge performance degradation. Numbers are now many times worse than before or 
after my optimizations. To me this shows that almost a complete rewrite (or 
revert) is needed.

-Petri


> -Original Message-
> From: lng-odp [mailto:lng-odp-boun...@lists.linaro.org] On Behalf Of Bill
> Fischofer
> Sent: Thursday, January 05, 2017 4:56 PM
> To: lng-odp@lists.linaro.org
> Subject: [lng-odp] [API-NEXT PATCHv6 2/5] linux-generic: packet: implement
> reference apis
> 
> Implement the APIs:
> - odp_packet_ref_static()
> - odp_packet_ref()
> - odp_packet_ref_pkt()
> - odp_packet_is_ref()
> - odp_packet_unshared_len()
> 
> This also involves functional upgrades to the existing packet manipulation
> APIs to work with packet references as input arguments.
> 
> Signed-off-by: Bill Fischofer 
> ---
>  .../linux-generic/include/odp_packet_internal.h|  73 ++-
>  platform/linux-generic/odp_packet.c| 542

[lng-odp] [Bug 2814] New: Failure in classification test when run with --enable-debug

[bugzilla-daemon]

https://bugs.linaro.org/show_bug.cgi?id=2814

Bug ID: 2814
   Summary: Failure in classification test when run with
--enable-debug
   Product: OpenDataPlane - linux- generic reference
   Version: api-next
  Hardware: Other
OS: Linux
Status: UNCONFIRMED
  Severity: enhancement
  Priority: ---
 Component: Buffers & Packets
  Assignee: bill.fischo...@linaro.org
  Reporter: bill.fischo...@linaro.org
CC: lng-odp@lists.linaro.org
  Target Milestone: ---

Classification fails when running with --enable-debug due to erroneous
ODP_ASSERT() in packet_set_len()

-- 
You are receiving this mail because:
You are on the CC list for the bug.

Re: [lng-odp] API next SIGABRT on classification test?

[Christophe Milard]

22b3986  - Bill Fischofer  : linux-generic: packet: implement reference apis

That is the one.

No need to open a bug?

On 12 January 2017 at 13:08, Bill Fischofer  wrote:
> On Thu, Jan 12, 2017 at 3:20 AM, Christophe Milard
>  wrote:
>> Am I alone seeing that?
>
> You found a bug that only surfaces if you're running with
> --enable-debug. It's a one-line fix that I'll post shortly. Thanks.
>
>>
>> PASS: validation/api/pktio/pktio_run.sh
>> SKIP: validation/api/pktio/pktio_run_tap.sh
>> PASS: validation/api/shmem/shmem_linux
>> PASS: ../../test/common_plat/validation/api/atomic/atomic_main
>> PASS: ../../test/common_plat/validation/api/barrier/barrier_main
>> PASS: ../../test/common_plat/validation/api/buffer/buffer_main
>> ../../test-driver: line 107: 17971 Aborted (core
>> dumped) "$@" > $log_file 2>&1
>> FAIL: 
>> ../../test/common_plat/validation/api/classification/classification_main
>> PASS: ../../test/common_plat/validation/api/cpumask/cpuma
>>
>> Gdb:
>> Suite: classification pmr tests
>>   Test: classification_test_pmr_term_tcp_dport
>> ...odp_packet_io.c:216:setup_pktio_entry():loop uses loop
>> _ishm.c:448:create_file():open failed for
>> /dev/hugepages/odp-17500-ishm-pool_ring_1: Permission denied.
>> _fdserver.c:277:_odp_fdserver_register_fd():FD client register:
>> pid=17500 key=10, fd=13
>> _fdserver.c:461:handle_request():storing {ctx=1, key=10}->fd=15
>> _ishm.c:448:create_file():open failed for
>> /dev/hugepages/odp-17500-ishm-DefaultPool: Permission denied.
>> _fdserver.c:277:_odp_fdserver_register_fd():FD client register:
>> pid=17500 key=11, fd=14
>> _fdserver.c:461:handle_request():storing {ctx=1, key=11}->fd=16
>> _ishm.c:448:create_file():open failed for
>> /dev/hugepages/odp-17500-ishm-pool_ring_2: Permission denied.
>> _fdserver.c:277:_odp_fdserver_register_fd():FD client register:
>> pid=17500 key=12, fd=15
>> _fdserver.c:461:handle_request():storing {ctx=1, key=12}->fd=17
>> _ishm.c:448:create_file():open failed for
>> /dev/hugepages/odp-17500-ishm-tcp_dport1: Permission denied.
>> _fdserver.c:277:_odp_fdserver_register_fd():FD client register:
>> pid=17500 key=13, fd=16
>> _fdserver.c:461:handle_request():storing {ctx=1, key=13}->fd=18
>> ./include/odp_packet_internal.h:327:packet_set_len():packet_ref_count(pkt_hdr)
>> == 1
>>
>> Program received signal SIGABRT, Aborted.
>> 0x76d89428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/l
>>
>> Christophe

[lng-odp] [Bug 2494] Extend odp_pktio_capability_t to include minimal pool size required by pktio implementation.

[bugzilla-daemon]

https://bugs.linaro.org/show_bug.cgi?id=2494

--- Comment #14 from Bala Manoharan  ---
Further internal design discussions on how to avoid this scenario.
If Maciej is fine this bug could be closed.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[lng-odp] [API-NEXT PATCH] linux-generic: packet: remove erroneous assert

[Bill Fischofer]

odp_packet_set_len() is only called from the classifier on paths for
which a dummy header may be used. Rather than setting a dummy reference
count to validate, it's simpler to just delete the assert. This resolves
Bug https://bugs.linaro.org/show_bug.cgi?id=2814

Signed-off-by: Bill Fischofer 
---
 platform/linux-generic/include/odp_packet_internal.h | 2 --
 1 file changed, 2 deletions(-)

diff --git a/platform/linux-generic/include/odp_packet_internal.h 
b/platform/linux-generic/include/odp_packet_internal.h
index 607560d..c5dc989 100644
--- a/platform/linux-generic/include/odp_packet_internal.h
+++ b/platform/linux-generic/include/odp_packet_internal.h
@@ -324,8 +324,6 @@ static inline void packet_ref_count_set(odp_packet_hdr_t 
*pkt_hdr, uint32_t n)
 
 static inline void packet_set_len(odp_packet_hdr_t *pkt_hdr, uint32_t len)
 {
-   ODP_ASSERT(packet_ref_count(pkt_hdr) == 1);
-
pkt_hdr->frame_len = len;
pkt_hdr->unshared_len = len;
 }
-- 
2.9.3

Re: [lng-odp] API next SIGABRT on classification test?

[Bill Fischofer]

I just opened Bug https://bugs.linaro.org/show_bug.cgi?id=2814 for
this and posted a patch to fix it.  Thanks.

On Thu, Jan 12, 2017 at 6:13 AM, Christophe Milard
 wrote:
> 22b3986  - Bill Fischofer  : linux-generic: packet: implement reference 
> apis
>
> That is the one.
>
> No need to open a bug?
>
> On 12 January 2017 at 13:08, Bill Fischofer  wrote:
>> On Thu, Jan 12, 2017 at 3:20 AM, Christophe Milard
>>  wrote:
>>> Am I alone seeing that?
>>
>> You found a bug that only surfaces if you're running with
>> --enable-debug. It's a one-line fix that I'll post shortly. Thanks.
>>
>>>
>>> PASS: validation/api/pktio/pktio_run.sh
>>> SKIP: validation/api/pktio/pktio_run_tap.sh
>>> PASS: validation/api/shmem/shmem_linux
>>> PASS: ../../test/common_plat/validation/api/atomic/atomic_main
>>> PASS: ../../test/common_plat/validation/api/barrier/barrier_main
>>> PASS: ../../test/common_plat/validation/api/buffer/buffer_main
>>> ../../test-driver: line 107: 17971 Aborted (core
>>> dumped) "$@" > $log_file 2>&1
>>> FAIL: 
>>> ../../test/common_plat/validation/api/classification/classification_main
>>> PASS: ../../test/common_plat/validation/api/cpumask/cpuma
>>>
>>> Gdb:
>>> Suite: classification pmr tests
>>>   Test: classification_test_pmr_term_tcp_dport
>>> ...odp_packet_io.c:216:setup_pktio_entry():loop uses loop
>>> _ishm.c:448:create_file():open failed for
>>> /dev/hugepages/odp-17500-ishm-pool_ring_1: Permission denied.
>>> _fdserver.c:277:_odp_fdserver_register_fd():FD client register:
>>> pid=17500 key=10, fd=13
>>> _fdserver.c:461:handle_request():storing {ctx=1, key=10}->fd=15
>>> _ishm.c:448:create_file():open failed for
>>> /dev/hugepages/odp-17500-ishm-DefaultPool: Permission denied.
>>> _fdserver.c:277:_odp_fdserver_register_fd():FD client register:
>>> pid=17500 key=11, fd=14
>>> _fdserver.c:461:handle_request():storing {ctx=1, key=11}->fd=16
>>> _ishm.c:448:create_file():open failed for
>>> /dev/hugepages/odp-17500-ishm-pool_ring_2: Permission denied.
>>> _fdserver.c:277:_odp_fdserver_register_fd():FD client register:
>>> pid=17500 key=12, fd=15
>>> _fdserver.c:461:handle_request():storing {ctx=1, key=12}->fd=17
>>> _ishm.c:448:create_file():open failed for
>>> /dev/hugepages/odp-17500-ishm-tcp_dport1: Permission denied.
>>> _fdserver.c:277:_odp_fdserver_register_fd():FD client register:
>>> pid=17500 key=13, fd=16
>>> _fdserver.c:461:handle_request():storing {ctx=1, key=13}->fd=18
>>> ./include/odp_packet_internal.h:327:packet_set_len():packet_ref_count(pkt_hdr)
>>> == 1
>>>
>>> Program received signal SIGABRT, Aborted.
>>> 0x76d89428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/l
>>>
>>> Christophe

Re: [lng-odp] API next SIGABRT on classification test?

[Christophe Milard]

OK. thanks for taking it from there :-)

On 12 January 2017 at 13:15, Bill Fischofer  wrote:
> I just opened Bug https://bugs.linaro.org/show_bug.cgi?id=2814 for
> this and posted a patch to fix it.  Thanks.
>
> On Thu, Jan 12, 2017 at 6:13 AM, Christophe Milard
>  wrote:
>> 22b3986  - Bill Fischofer  : linux-generic: packet: implement reference 
>> apis
>>
>> That is the one.
>>
>> No need to open a bug?
>>
>> On 12 January 2017 at 13:08, Bill Fischofer  
>> wrote:
>>> On Thu, Jan 12, 2017 at 3:20 AM, Christophe Milard
>>>  wrote:
 Am I alone seeing that?
>>>
>>> You found a bug that only surfaces if you're running with
>>> --enable-debug. It's a one-line fix that I'll post shortly. Thanks.
>>>

 PASS: validation/api/pktio/pktio_run.sh
 SKIP: validation/api/pktio/pktio_run_tap.sh
 PASS: validation/api/shmem/shmem_linux
 PASS: ../../test/common_plat/validation/api/atomic/atomic_main
 PASS: ../../test/common_plat/validation/api/barrier/barrier_main
 PASS: ../../test/common_plat/validation/api/buffer/buffer_main
 ../../test-driver: line 107: 17971 Aborted (core
 dumped) "$@" > $log_file 2>&1
 FAIL: 
 ../../test/common_plat/validation/api/classification/classification_main
 PASS: ../../test/common_plat/validation/api/cpumask/cpuma

 Gdb:
 Suite: classification pmr tests
   Test: classification_test_pmr_term_tcp_dport
 ...odp_packet_io.c:216:setup_pktio_entry():loop uses loop
 _ishm.c:448:create_file():open failed for
 /dev/hugepages/odp-17500-ishm-pool_ring_1: Permission denied.
 _fdserver.c:277:_odp_fdserver_register_fd():FD client register:
 pid=17500 key=10, fd=13
 _fdserver.c:461:handle_request():storing {ctx=1, key=10}->fd=15
 _ishm.c:448:create_file():open failed for
 /dev/hugepages/odp-17500-ishm-DefaultPool: Permission denied.
 _fdserver.c:277:_odp_fdserver_register_fd():FD client register:
 pid=17500 key=11, fd=14
 _fdserver.c:461:handle_request():storing {ctx=1, key=11}->fd=16
 _ishm.c:448:create_file():open failed for
 /dev/hugepages/odp-17500-ishm-pool_ring_2: Permission denied.
 _fdserver.c:277:_odp_fdserver_register_fd():FD client register:
 pid=17500 key=12, fd=15
 _fdserver.c:461:handle_request():storing {ctx=1, key=12}->fd=17
 _ishm.c:448:create_file():open failed for
 /dev/hugepages/odp-17500-ishm-tcp_dport1: Permission denied.
 _fdserver.c:277:_odp_fdserver_register_fd():FD client register:
 pid=17500 key=13, fd=16
 _fdserver.c:461:handle_request():storing {ctx=1, key=13}->fd=18
 ./include/odp_packet_internal.h:327:packet_set_len():packet_ref_count(pkt_hdr)
 == 1

 Program received signal SIGABRT, Aborted.
 0x76d89428 in __GI_raise (sig=sig@entry=6) at 
 ../sysdeps/unix/sysv/l

 Christophe

Re: [lng-odp] [API-NEXT PATCHv6 2/5] linux-generic: packet: implement reference apis

[Bill Fischofer]

On Thu, Jan 12, 2017 at 6:12 AM, Savolainen, Petri (Nokia - FI/Espoo)
 wrote:
> This patch is now merged, although I had some doubts that it has bad impact 
> on performance. Here are some performance results for couple of simple, 
> single thread packet alloc/free test cases. No references involved, just 
> plain packets as before.
>
> Test results before and after "linux-generic: packet: implement reference 
> apis":
>
> CPU cycles per operation
> before  after
> packet_alloc127.8   250.9   +96 %
> packet_alloc_multi  873.7   1538.8  +76 %
> packet_free 31  116.8   +277 %
> packet_free_multi   214.5   1369.2  +538 %
> packet_alloc_free   73.4193.7   +164 %
> packet_alloc_free_multi 286.1   1228.8  +330 %
>
>
> Huge performance degradation. Numbers are now many times worse than before or 
> after my optimizations. To me this shows that almost a complete rewrite (or 
> revert) is needed.

My guess is this is due to the atomics needed for reference counting
not being properly inlined internally. I know you did similar
optimizations for ticketlocks. Let me look into this and post a patch
that does the same for the atomics.

>
> -Petri
>
>
>> -Original Message-
>> From: lng-odp [mailto:lng-odp-boun...@lists.linaro.org] On Behalf Of Bill
>> Fischofer
>> Sent: Thursday, January 05, 2017 4:56 PM
>> To: lng-odp@lists.linaro.org
>> Subject: [lng-odp] [API-NEXT PATCHv6 2/5] linux-generic: packet: implement
>> reference apis
>>
>> Implement the APIs:
>> - odp_packet_ref_static()
>> - odp_packet_ref()
>> - odp_packet_ref_pkt()
>> - odp_packet_is_ref()
>> - odp_packet_unshared_len()
>>
>> This also involves functional upgrades to the existing packet manipulation
>> APIs to work with packet references as input arguments.
>>
>> Signed-off-by: Bill Fischofer 
>> ---
>>  .../linux-generic/include/odp_packet_internal.h|  73 ++-
>>  platform/linux-generic/odp_packet.c| 542

Re: [lng-odp] [API-NEXT PATCH] linux-generic: packet: remove erroneous assert

[Christophe Milard]

On 12 January 2017 at 13:14, Bill Fischofer  wrote:
> odp_packet_set_len() is only called from the classifier on paths for
> which a dummy header may be used. Rather than setting a dummy reference
> count to validate, it's simpler to just delete the assert. This resolves
> Bug https://bugs.linaro.org/show_bug.cgi?id=2814
>
> Signed-off-by: Bill Fischofer 

Reviewed-by: Christophe Milard 

> ---
>  platform/linux-generic/include/odp_packet_internal.h | 2 --
>  1 file changed, 2 deletions(-)
>
> diff --git a/platform/linux-generic/include/odp_packet_internal.h 
> b/platform/linux-generic/include/odp_packet_internal.h
> index 607560d..c5dc989 100644
> --- a/platform/linux-generic/include/odp_packet_internal.h
> +++ b/platform/linux-generic/include/odp_packet_internal.h
> @@ -324,8 +324,6 @@ static inline void packet_ref_count_set(odp_packet_hdr_t 
> *pkt_hdr, uint32_t n)
>
>  static inline void packet_set_len(odp_packet_hdr_t *pkt_hdr, uint32_t len)
>  {
> -   ODP_ASSERT(packet_ref_count(pkt_hdr) == 1);
> -
> pkt_hdr->frame_len = len;
> pkt_hdr->unshared_len = len;
>  }
> --
> 2.9.3
>

Re: [lng-odp] [API-NEXT PATCHv6 2/5] linux-generic: packet: implement reference apis

[Savolainen, Petri (Nokia - FI/Espoo)]

> -Original Message-
> From: Bill Fischofer [mailto:bill.fischo...@linaro.org]
> Sent: Thursday, January 12, 2017 2:22 PM
> To: Savolainen, Petri (Nokia - FI/Espoo)  labs.com>
> Cc: lng-odp@lists.linaro.org
> Subject: Re: [lng-odp] [API-NEXT PATCHv6 2/5] linux-generic: packet:
> implement reference apis
> 
> On Thu, Jan 12, 2017 at 6:12 AM, Savolainen, Petri (Nokia - FI/Espoo)
>  wrote:
> > This patch is now merged, although I had some doubts that it has bad
> impact on performance. Here are some performance results for couple of
> simple, single thread packet alloc/free test cases. No references
> involved, just plain packets as before.
> >
> > Test results before and after "linux-generic: packet: implement
> reference apis":
> >
> > CPU cycles per operation
> > before  after
> > packet_alloc127.8   250.9   +96 %
> > packet_alloc_multi  873.7   1538.8  +76 %
> > packet_free 31  116.8   +277 %
> > packet_free_multi   214.5   1369.2  +538 %
> > packet_alloc_free   73.4193.7   +164 %
> > packet_alloc_free_multi 286.1   1228.8  +330 %
> >
> >
> > Huge performance degradation. Numbers are now many times worse than
> before or after my optimizations. To me this shows that almost a complete
> rewrite (or revert) is needed.
> 
> My guess is this is due to the atomics needed for reference counting
> not being properly inlined internally. I know you did similar
> optimizations for ticketlocks. Let me look into this and post a patch
> that does the same for the atomics.

Atomics are inlined already. Also atomic operations should not be required if 
there's a single refence.

-Petri

Re: [lng-odp] [API-NEXT PATCHv6 2/5] linux-generic: packet: implement reference apis

[Bill Fischofer]

On Thu, Jan 12, 2017 at 6:27 AM, Savolainen, Petri (Nokia - FI/Espoo)
 wrote:
>
>
>> -Original Message-
>> From: Bill Fischofer [mailto:bill.fischo...@linaro.org]
>> Sent: Thursday, January 12, 2017 2:22 PM
>> To: Savolainen, Petri (Nokia - FI/Espoo) > labs.com>
>> Cc: lng-odp@lists.linaro.org
>> Subject: Re: [lng-odp] [API-NEXT PATCHv6 2/5] linux-generic: packet:
>> implement reference apis
>>
>> On Thu, Jan 12, 2017 at 6:12 AM, Savolainen, Petri (Nokia - FI/Espoo)
>>  wrote:
>> > This patch is now merged, although I had some doubts that it has bad
>> impact on performance. Here are some performance results for couple of
>> simple, single thread packet alloc/free test cases. No references
>> involved, just plain packets as before.
>> >
>> > Test results before and after "linux-generic: packet: implement
>> reference apis":
>> >
>> > CPU cycles per operation
>> > before  after
>> > packet_alloc127.8   250.9   +96 %
>> > packet_alloc_multi  873.7   1538.8  +76 %
>> > packet_free 31  116.8   +277 %
>> > packet_free_multi   214.5   1369.2  +538 %
>> > packet_alloc_free   73.4193.7   +164 %
>> > packet_alloc_free_multi 286.1   1228.8  +330 %
>> >
>> >
>> > Huge performance degradation. Numbers are now many times worse than
>> before or after my optimizations. To me this shows that almost a complete
>> rewrite (or revert) is needed.
>>
>> My guess is this is due to the atomics needed for reference counting
>> not being properly inlined internally. I know you did similar
>> optimizations for ticketlocks. Let me look into this and post a patch
>> that does the same for the atomics.
>
> Atomics are inlined already. Also atomic operations should not be required if 
> there's a single refence.

The ref_count still needs to be initialized on alloc and decremented
on free to see if the packet should really be freed. The only way to
know whether you're dealing with a single reference or not is to
check. Do you have a breakdown of where the hotspots are? These
numbers do seem high.

>
> -Petri
>
>

Re: [lng-odp] [API-NEXT PATCHv6 2/5] linux-generic: packet: implement reference apis

[Savolainen, Petri (Nokia - FI/Espoo)]

> >> > Huge performance degradation. Numbers are now many times worse than
> >> before or after my optimizations. To me this shows that almost a
> complete
> >> rewrite (or revert) is needed.
> >>
> >> My guess is this is due to the atomics needed for reference counting
> >> not being properly inlined internally. I know you did similar
> >> optimizations for ticketlocks. Let me look into this and post a patch
> >> that does the same for the atomics.
> >
> > Atomics are inlined already. Also atomic operations should not be
> required if there's a single refence.
> 
> The ref_count still needs to be initialized on alloc and decremented
> on free to see if the packet should really be freed. The only way to
> know whether you're dealing with a single reference or not is to
> check. Do you have a breakdown of where the hotspots are? These
> numbers do seem high.
> 

No, I didn't look into it any deeper.

-Petri

Re: [lng-odp] [API-NEXT PATCHv6 2/5] linux-generic: packet: implement reference apis

[Bill Fischofer]

On Thu, Jan 12, 2017 at 6:52 AM, Savolainen, Petri (Nokia - FI/Espoo)
 wrote:
>
>> >> > Huge performance degradation. Numbers are now many times worse than
>> >> before or after my optimizations. To me this shows that almost a
>> complete
>> >> rewrite (or revert) is needed.
>> >>
>> >> My guess is this is due to the atomics needed for reference counting
>> >> not being properly inlined internally. I know you did similar
>> >> optimizations for ticketlocks. Let me look into this and post a patch
>> >> that does the same for the atomics.
>> >
>> > Atomics are inlined already. Also atomic operations should not be
>> required if there's a single refence.
>>
>> The ref_count still needs to be initialized on alloc and decremented
>> on free to see if the packet should really be freed. The only way to
>> know whether you're dealing with a single reference or not is to
>> check. Do you have a breakdown of where the hotspots are? These
>> numbers do seem high.
>>
>
> No, I didn't look into it any deeper.

If we look at the comparative pathlength for packet_alloc(), the only
changes are to the internal routines packet_init() and
init_segments(). The rest of the code is identical.

For packet_init() the delta is these added two lines:

/* By default packet has no references */
pkt_hdr->unshared_len = len;
pkt_hdr->ref_hdr = NULL;

For init_segments() the delta is similarly tiny:

packet_ref_count_set(hdr, 1);

Where this is an inline function that expands to
odp_atomic_init_u32(&hdr->ref_count, 1);

The changes on the packet_free() side are similarly tiny. Basically we
decrement the ref_count and only actually free the buffer if the
ref_count is now zero.

I don't see how these translate into a near doubling of cycles for
these tests. It seems there must be something else going on.

>
> -Petri

[lng-odp] [Bug 2814] Failure in classification test when run with --enable-debug

[bugzilla-daemon]

https://bugs.linaro.org/show_bug.cgi?id=2814

Bill Fischofer  changed:

   What|Removed |Added

 Status|UNCONFIRMED |IN_PROGRESS
 Ever confirmed|0   |1

--- Comment #1 from Bill Fischofer  ---
Patch http://patches.opendataplane.org/patch/7853/ posted to address this
issue.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

Re: [lng-odp] [API-NEXT PATCH] linux-generic: packet: remove erroneous assert

[Maxim Uvarov]

Merged,
Maxim.

On 01/12/17 15:27, Christophe Milard wrote:
> On 12 January 2017 at 13:14, Bill Fischofer  wrote:
>> odp_packet_set_len() is only called from the classifier on paths for
>> which a dummy header may be used. Rather than setting a dummy reference
>> count to validate, it's simpler to just delete the assert. This resolves
>> Bug https://bugs.linaro.org/show_bug.cgi?id=2814
>>
>> Signed-off-by: Bill Fischofer 
> 
> Reviewed-by: Christophe Milard 
> 
>> ---
>>  platform/linux-generic/include/odp_packet_internal.h | 2 --
>>  1 file changed, 2 deletions(-)
>>
>> diff --git a/platform/linux-generic/include/odp_packet_internal.h 
>> b/platform/linux-generic/include/odp_packet_internal.h
>> index 607560d..c5dc989 100644
>> --- a/platform/linux-generic/include/odp_packet_internal.h
>> +++ b/platform/linux-generic/include/odp_packet_internal.h
>> @@ -324,8 +324,6 @@ static inline void packet_ref_count_set(odp_packet_hdr_t 
>> *pkt_hdr, uint32_t n)
>>
>>  static inline void packet_set_len(odp_packet_hdr_t *pkt_hdr, uint32_t len)
>>  {
>> -   ODP_ASSERT(packet_ref_count(pkt_hdr) == 1);
>> -
>> pkt_hdr->frame_len = len;
>> pkt_hdr->unshared_len = len;
>>  }
>> --
>> 2.9.3
>>

[lng-odp] [API-NEXT PATCH 1/2] linux-gen: ipsec: add capability function

[Matias Elo]

Add IPsec capability function (groundwork for a proper IPsec
implementation).

Signed-off-by: Matias Elo 
---
 platform/linux-generic/Makefile.am |  1 +
 platform/linux-generic/odp_ipsec.c | 16 
 2 files changed, 17 insertions(+)
 create mode 100644 platform/linux-generic/odp_ipsec.c

diff --git a/platform/linux-generic/Makefile.am 
b/platform/linux-generic/Makefile.am
index 999a7f5..f6c51a3 100644
--- a/platform/linux-generic/Makefile.am
+++ b/platform/linux-generic/Makefile.am
@@ -185,6 +185,7 @@ __LIB__libodp_linux_la_SOURCES = \
   odp_hash.c \
   odp_init.c \
   odp_impl.c \
+  odp_ipsec.c \
   odp_name_table.c \
   odp_packet.c \
   odp_packet_flags.c \
diff --git a/platform/linux-generic/odp_ipsec.c 
b/platform/linux-generic/odp_ipsec.c
new file mode 100644
index 000..ce7fded
--- /dev/null
+++ b/platform/linux-generic/odp_ipsec.c
@@ -0,0 +1,16 @@
+/* Copyright (c) 2017, Linaro Limited
+ * All rights reserved.
+ *
+ * SPDX-License-Identifier: BSD-3-Clause
+ */
+
+#include 
+
+#include 
+
+int odp_ipsec_capability(odp_ipsec_capability_t *capa)
+{
+   memset(capa, 0, sizeof(odp_ipsec_capability_t));
+
+   return 0;
+}
-- 
2.7.4

[lng-odp] [API-NEXT PATCH 2/2] validation: ipsec: add test for odp_ipsec_capability()

[Matias Elo]

Signed-off-by: Matias Elo 
---
 test/common_plat/m4/configure.m4   |  1 +
 test/common_plat/validation/api/Makefile.am|  1 +
 test/common_plat/validation/api/ipsec/.gitignore   |  1 +
 test/common_plat/validation/api/ipsec/Makefile.am  | 10 +
 test/common_plat/validation/api/ipsec/ipsec.c  | 44 ++
 test/common_plat/validation/api/ipsec/ipsec.h  | 24 
 test/common_plat/validation/api/ipsec/ipsec_main.c | 12 ++
 test/linux-generic/Makefile.am |  1 +
 8 files changed, 94 insertions(+)
 create mode 100644 test/common_plat/validation/api/ipsec/.gitignore
 create mode 100644 test/common_plat/validation/api/ipsec/Makefile.am
 create mode 100644 test/common_plat/validation/api/ipsec/ipsec.c
 create mode 100644 test/common_plat/validation/api/ipsec/ipsec.h
 create mode 100644 test/common_plat/validation/api/ipsec/ipsec_main.c

diff --git a/test/common_plat/m4/configure.m4 b/test/common_plat/m4/configure.m4
index 1fc350d..13a13bd 100644
--- a/test/common_plat/m4/configure.m4
+++ b/test/common_plat/m4/configure.m4
@@ -17,6 +17,7 @@ AC_CONFIG_FILES([test/common_plat/Makefile
 test/common_plat/validation/api/errno/Makefile
 test/common_plat/validation/api/hash/Makefile
 test/common_plat/validation/api/init/Makefile
+test/common_plat/validation/api/ipsec/Makefile
 test/common_plat/validation/api/lock/Makefile
 test/common_plat/validation/api/packet/Makefile
 test/common_plat/validation/api/pktio/Makefile
diff --git a/test/common_plat/validation/api/Makefile.am 
b/test/common_plat/validation/api/Makefile.am
index e2d30a6..0bab4c4 100644
--- a/test/common_plat/validation/api/Makefile.am
+++ b/test/common_plat/validation/api/Makefile.am
@@ -7,6 +7,7 @@ ODP_MODULES = atomic \
  errno \
  hash \
  init \
+ ipsec \
  lock \
  queue \
  packet \
diff --git a/test/common_plat/validation/api/ipsec/.gitignore 
b/test/common_plat/validation/api/ipsec/.gitignore
new file mode 100644
index 000..2def047
--- /dev/null
+++ b/test/common_plat/validation/api/ipsec/.gitignore
@@ -0,0 +1 @@
+ipsec_main
diff --git a/test/common_plat/validation/api/ipsec/Makefile.am 
b/test/common_plat/validation/api/ipsec/Makefile.am
new file mode 100644
index 000..106b8dc
--- /dev/null
+++ b/test/common_plat/validation/api/ipsec/Makefile.am
@@ -0,0 +1,10 @@
+include ../Makefile.inc
+
+noinst_LTLIBRARIES = libtestipsec.la
+libtestipsec_la_SOURCES = ipsec.c
+
+test_PROGRAMS = ipsec_main$(EXEEXT)
+dist_ipsec_main_SOURCES = ipsec_main.c
+ipsec_main_LDADD = libtestipsec.la $(LIBCUNIT_COMMON) $(LIBODP)
+
+EXTRA_DIST = ipsec.h
diff --git a/test/common_plat/validation/api/ipsec/ipsec.c 
b/test/common_plat/validation/api/ipsec/ipsec.c
new file mode 100644
index 000..7834803
--- /dev/null
+++ b/test/common_plat/validation/api/ipsec/ipsec.c
@@ -0,0 +1,44 @@
+/* Copyright (c) 2017, Linaro Limited
+ * All rights reserved.
+ *
+ * SPDX-License-Identifier: BSD-3-Clause
+ */
+
+#include 
+#include 
+#include 
+
+#include "ipsec.h"
+
+void ipsec_test_capability(void)
+{
+   odp_ipsec_capability_t capa;
+
+   CU_ASSERT(odp_ipsec_capability(&capa) == 0);
+}
+
+odp_testinfo_t ipsec_suite[] = {
+   ODP_TEST_INFO(ipsec_test_capability),
+   ODP_TEST_INFO_NULL
+};
+
+odp_suiteinfo_t ipsec_suites[] = {
+   {"IPsec", NULL, NULL, ipsec_suite},
+   ODP_SUITE_INFO_NULL,
+};
+
+int ipsec_main(int argc, char *argv[])
+{
+   int ret;
+
+   /* parse common options: */
+   if (odp_cunit_parse_options(argc, argv))
+   return -1;
+
+   ret = odp_cunit_register(ipsec_suites);
+
+   if (ret == 0)
+   ret = odp_cunit_run();
+
+   return ret;
+}
diff --git a/test/common_plat/validation/api/ipsec/ipsec.h 
b/test/common_plat/validation/api/ipsec/ipsec.h
new file mode 100644
index 000..290a186
--- /dev/null
+++ b/test/common_plat/validation/api/ipsec/ipsec.h
@@ -0,0 +1,24 @@
+/* Copyright (c) 2017, Linaro Limited
+ * All rights reserved.
+ *
+ * SPDX-License-Identifier: BSD-3-Clause
+ */
+
+#ifndef _ODP_TEST_IPSEC_H_
+#define _ODP_TEST_IPSEC_H_
+
+#include 
+
+/* test functions: */
+void ipsec_test_capability(void);
+
+/* test arrays: */
+extern odp_testinfo_t ipsec_suite[];
+
+/* test registry: */
+extern odp_suiteinfo_t ipsec_suites[];
+
+/* main test program: */
+int ipsec_main(int argc, char *argv[]);
+
+#endif
diff --git a/test/common_plat/validation/api/ipsec/ipsec_main.c 
b/test/common_plat/validation/api/ipsec/ipsec_main.c
new file mode 100644
index 000..63a7a55
--- /dev/null
+++ b/test/common_plat/validation/api/ipsec/ipsec_main.c
@@ -0,0 +1,12 @@
+/* Copyright (c) 2017, Linaro Limited
+ * All rights reserved.
+ *
+ * SPDX-License-Identifier: BSD-3-Clause
+ */
+
+#include "ipsec.h"
+
+int main(int argc, 

Re: [lng-odp] [API-NEXT PATCHv6 2/5] linux-generic: packet: implement reference apis

[Bill Fischofer]

On Thu, Jan 12, 2017 at 6:58 AM, Bill Fischofer
 wrote:
> On Thu, Jan 12, 2017 at 6:52 AM, Savolainen, Petri (Nokia - FI/Espoo)
>  wrote:
>>
>>> >> > Huge performance degradation. Numbers are now many times worse than
>>> >> before or after my optimizations. To me this shows that almost a
>>> complete
>>> >> rewrite (or revert) is needed.
>>> >>
>>> >> My guess is this is due to the atomics needed for reference counting
>>> >> not being properly inlined internally. I know you did similar
>>> >> optimizations for ticketlocks. Let me look into this and post a patch
>>> >> that does the same for the atomics.
>>> >
>>> > Atomics are inlined already. Also atomic operations should not be
>>> required if there's a single refence.
>>>
>>> The ref_count still needs to be initialized on alloc and decremented
>>> on free to see if the packet should really be freed. The only way to
>>> know whether you're dealing with a single reference or not is to
>>> check. Do you have a breakdown of where the hotspots are? These
>>> numbers do seem high.
>>>
>>
>> No, I didn't look into it any deeper.
>
> If we look at the comparative pathlength for packet_alloc(), the only
> changes are to the internal routines packet_init() and
> init_segments(). The rest of the code is identical.
>
> For packet_init() the delta is these added two lines:
>
> /* By default packet has no references */
> pkt_hdr->unshared_len = len;
> pkt_hdr->ref_hdr = NULL;
>
> For init_segments() the delta is similarly tiny:
>
> packet_ref_count_set(hdr, 1);
>
> Where this is an inline function that expands to
> odp_atomic_init_u32(&hdr->ref_count, 1);
>
> The changes on the packet_free() side are similarly tiny. Basically we
> decrement the ref_count and only actually free the buffer if the
> ref_count is now zero.
>
> I don't see how these translate into a near doubling of cycles for
> these tests. It seems there must be something else going on.

I think I see some potential larger deltas in the packet free path.
I'll dig into this further and post a patch .

>
>>
>> -Petri

Re: [lng-odp] [API-NEXT PATCHv3 3/4] linux-gen: add generic bitmaps and iterators

[Maxim Uvarov]

On 01/12/17 04:17, Yi He wrote:
> Yes, ARRAY_SIZE is a kernel macro, I looked into the checkpatch.pl
>  and see no flag to enable/disable this check,
> shall we comment it out or remove that piece of code?
> 
> Best Regards, Yi

yes, that warning is ok.

Maxim.


> 
> On 12 January 2017 at 07:40, Bill Fischofer  > wrote:
> 
> On Wed, Jan 11, 2017 at 1:50 AM, Yi He  > wrote:
> > Add C++ template alike bitmap to allow
> > instantiate bitmap data structure of any size,
> > and provide iterators to help walking through
> > the bitmap objects.
> >
> > Signed-off-by: Yi He mailto:yi...@linaro.org>>
> > ---
> >  platform/linux-generic/Makefile.am |   2 +
> >  .../linux-generic/include/odp_bitmap_internal.h| 317
> +
> >  platform/linux-generic/odp_bitmap.c| 315
> 
> >  3 files changed, 634 insertions(+)
> >  create mode 100644
> platform/linux-generic/include/odp_bitmap_internal.h
> >  create mode 100644 platform/linux-generic/odp_bitmap.c
> >
> > diff --git a/platform/linux-generic/Makefile.am
> b/platform/linux-generic/Makefile.am
> > index 999a7f5..5b38c7b 100644
> > --- a/platform/linux-generic/Makefile.am
> > +++ b/platform/linux-generic/Makefile.am
> > @@ -130,6 +130,7 @@ noinst_HEADERS = \
> >   ${srcdir}/include/odp_align_internal.h \
> >   ${srcdir}/include/odp_atomic_internal.h \
> >   ${srcdir}/include/odp_buffer_inlines.h \
> > + ${srcdir}/include/odp_bitmap_internal.h \
> >   ${srcdir}/include/odp_buffer_internal.h \
> >   ${srcdir}/include/odp_classification_datamodel.h \
> >   ${srcdir}/include/odp_classification_inlines.h \
> > @@ -173,6 +174,7 @@ __LIB__libodp_linux_la_SOURCES = \
> >_ishmphy.c \
> >odp_atomic.c \
> >odp_barrier.c \
> > +  odp_bitmap.c \
> >odp_buffer.c \
> >odp_byteorder.c \
> >odp_classification.c \
> > diff --git a/platform/linux-generic/include/odp_bitmap_internal.h
> b/platform/linux-generic/include/odp_bitmap_internal.h
> > new file mode 100644
> > index 000..1be4d02
> > --- /dev/null
> > +++ b/platform/linux-generic/include/odp_bitmap_internal.h
> > @@ -0,0 +1,317 @@
> > +/* Copyright (c) 2016, Linaro Limited
> > + * All rights reserved.
> > + *
> > + * SPDX-License-Identifier: BSD-3-Clause
> > + */
> > +
> > +/**
> > + * @file
> > + *
> > + * ODP generic bitmap types and operations.
> > + */
> > +
> > +#ifndef ODP_BITMAP_INTERNAL_H_
> > +#define ODP_BITMAP_INTERNAL_H_
> > +
> > +#ifdef __cplusplus
> > +extern "C" {
> > +#endif
> > +
> > +#include 
> > +#include 
> > +#include 
> > +#include 
> > +
> > +/* Generate unique identifier for instantiated class */
> > +#define TOKENIZE(template, line) \
> > +   template ## _ ## line ## _ ## __COUNTER__
> > +
> > +/* Array size in general */
> > +#define ARRAY_SIZE(array) (sizeof(array) / sizeof(array[0]))
> 
> Checkpatch flags this with the following:
> ---
> WARNING: Prefer ARRAY_SIZE(array)
> #76: FILE: platform/linux-generic/include/odp_bitmap_internal.h:30:
> +#define ARRAY_SIZE(array) (sizeof(array) / sizeof(array[0]))
> 
> total: 0 errors, 1 warnings, 0 checks, 646 lines checked
> 
> NOTE: Ignored message types: BIT_MACRO COMPARISON_TO_NULL
> DEPRECATED_VARIABLE NEW_TYPEDEFS SPLIT_STRING SSCANF_TO_KSTRTO
> 
> 0003-linux-gen-add-generic-bitmaps-and-iterators.patch has style
> problems, please review.
> ---
> 
> I suspect that ARRAY_SIZE is a Linux internal macro, in which case our
> internal checkpatch rules may need updating.
> > +
> > +#define BITS_PER_BYTE  (8)
> > +#define BITS_PER_LONG  __WORDSIZE
> > +#define BYTES_PER_LONG (BITS_PER_LONG / BITS_PER_BYTE)
> > +
> > +#define BIT_WORD(nr)   ((nr) / BITS_PER_LONG)
> > +#define BITS_TO_LONGS(nr) BIT_WORD(nr + BITS_PER_LONG - 1)
> > +
> > +#define BITMAP_FIRST_WORD_MASK(start) \
> > +   (~0UL << ((start) & (BITS_PER_LONG - 1)))
> > +#define BITMAP_LAST_WORD_MASK(nbits)  \
> > +   (~0UL >> (-(nbits) & (BITS_PER_LONG - 1)))
> > +
> > +/* WAPL bitmap base class */
> > +typedef struct {
> > +   unsigned int  nwords;
> > +   unsigned int  *pl;
> > +   unsigned long *ul;
> > +} wapl_bitmap_t;
> > +
> > +/*
> > + * Word-Aligned Position List (WAPL) bitmap, w

[lng-odp] [Bug 2814] Failure in classification test when run with --enable-debug

[bugzilla-daemon]

https://bugs.linaro.org/show_bug.cgi?id=2814

Mike Holmes  changed:

   What|Removed |Added

 CC||mike.hol...@linaro.org
 Resolution|--- |FIXED
 Status|IN_PROGRESS |RESOLVED

--- Comment #2 from Mike Holmes  ---
c28c81745896d433262cc964d341f6e2d448c12f

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[lng-odp] [Bug 2812] helper/test/process fails on a single core system

[bugzilla-daemon]

https://bugs.linaro.org/show_bug.cgi?id=2812

Mike Holmes  changed:

   What|Removed |Added

 Ever confirmed|0   |1
 Status|UNCONFIRMED |CONFIRMED
 CC||mike.hol...@linaro.org

-- 
You are receiving this mail because:
You are on the CC list for the bug.

Re: [lng-odp] [API-NEXT PATCHv3 1/4] linux-gen: sched: solve ordered context inversion

[Maxim Uvarov]

Merged,
Maxim.

On 01/12/17 02:41, Bill Fischofer wrote:
> For this series:
> 
> Reviewed-and-tested-by: Bill Fischofer 
> 
> On Wed, Jan 11, 2017 at 1:50 AM, Yi He  wrote:
>> For ordered queue, a thread consumes events (dequeue) and
>> acquires its unique sequential context in two steps, non
>> atomic and preemptable.
>>
>> This leads to potential ordered context inversion in case
>> the thread consumes prior events acquired subsequent context,
>> while the thread consumes subsequent events but acquired
>> prior context.
>>
>> This patch insert the ordered context acquisition into
>> event dequeue operation to make these two steps atomic.
>>
>> Signed-off-by: Yi He 
>> ---
>>  platform/linux-generic/include/odp_schedule_if.h | 3 +++
>>  platform/linux-generic/odp_queue.c   | 3 +++
>>  platform/linux-generic/odp_schedule.c| 7 ++-
>>  platform/linux-generic/odp_schedule_sp.c | 7 ++-
>>  4 files changed, 18 insertions(+), 2 deletions(-)
>>
>> diff --git a/platform/linux-generic/include/odp_schedule_if.h 
>> b/platform/linux-generic/include/odp_schedule_if.h
>> index 6c2b050..c0aee42 100644
>> --- a/platform/linux-generic/include/odp_schedule_if.h
>> +++ b/platform/linux-generic/include/odp_schedule_if.h
>> @@ -12,6 +12,7 @@ extern "C" {
>>  #endif
>>
>>  #include 
>> +#include 
>>  #include 
>>
>>  typedef void (*schedule_pktio_start_fn_t)(int pktio_index, int num_in_queue,
>> @@ -33,6 +34,7 @@ typedef int (*schedule_term_local_fn_t)(void);
>>  typedef void (*schedule_order_lock_fn_t)(void);
>>  typedef void (*schedule_order_unlock_fn_t)(void);
>>  typedef unsigned (*schedule_max_ordered_locks_fn_t)(void);
>> +typedef void (*schedule_save_context_fn_t)(queue_entry_t *queue);
>>
>>  typedef struct schedule_fn_t {
>> schedule_pktio_start_fn_t   pktio_start;
>> @@ -50,6 +52,7 @@ typedef struct schedule_fn_t {
>> schedule_order_lock_fn_torder_lock;
>> schedule_order_unlock_fn_t  order_unlock;
>> schedule_max_ordered_locks_fn_t max_ordered_locks;
>> +   schedule_save_context_fn_t  save_context;
>>  } schedule_fn_t;
>>
>>  /* Interface towards the scheduler */
>> diff --git a/platform/linux-generic/odp_queue.c 
>> b/platform/linux-generic/odp_queue.c
>> index d9cb9f3..3b6a68b 100644
>> --- a/platform/linux-generic/odp_queue.c
>> +++ b/platform/linux-generic/odp_queue.c
>> @@ -568,6 +568,9 @@ static inline int deq_multi(queue_entry_t *queue, 
>> odp_buffer_hdr_t *buf_hdr[],
>> if (hdr == NULL)
>> queue->s.tail = NULL;
>>
>> +   if (queue->s.type == ODP_QUEUE_TYPE_SCHED)
>> +   sched_fn->save_context(queue);
>> +
>> UNLOCK(&queue->s.lock);
>>
>> return i;
>> diff --git a/platform/linux-generic/odp_schedule.c 
>> b/platform/linux-generic/odp_schedule.c
>> index 645630a..264c58f 100644
>> --- a/platform/linux-generic/odp_schedule.c
>> +++ b/platform/linux-generic/odp_schedule.c
>> @@ -1205,6 +1205,10 @@ static int schedule_num_grps(void)
>> return NUM_SCHED_GRPS;
>>  }
>>
>> +static void schedule_save_context(queue_entry_t *queue ODP_UNUSED)
>> +{
>> +}
>> +
>>  /* Fill in scheduler interface */
>>  const schedule_fn_t schedule_default_fn = {
>> .pktio_start = schedule_pktio_start,
>> @@ -1221,7 +1225,8 @@ const schedule_fn_t schedule_default_fn = {
>> .term_local  = schedule_term_local,
>> .order_lock = order_lock,
>> .order_unlock = order_unlock,
>> -   .max_ordered_locks = schedule_max_ordered_locks
>> +   .max_ordered_locks = schedule_max_ordered_locks,
>> +   .save_context = schedule_save_context
>>  };
>>
>>  /* Fill in scheduler API calls */
>> diff --git a/platform/linux-generic/odp_schedule_sp.c 
>> b/platform/linux-generic/odp_schedule_sp.c
>> index 5150d28..1406312 100644
>> --- a/platform/linux-generic/odp_schedule_sp.c
>> +++ b/platform/linux-generic/odp_schedule_sp.c
>> @@ -803,6 +803,10 @@ static void order_unlock(void)
>>  {
>>  }
>>
>> +static void save_context(queue_entry_t *queue ODP_UNUSED)
>> +{
>> +}
>> +
>>  /* Fill in scheduler interface */
>>  const schedule_fn_t schedule_sp_fn = {
>> .pktio_start   = pktio_start,
>> @@ -819,7 +823,8 @@ const schedule_fn_t schedule_sp_fn = {
>> .term_local= term_local,
>> .order_lock =order_lock,
>> .order_unlock =  order_unlock,
>> -   .max_ordered_locks = max_ordered_locks
>> +   .max_ordered_locks = max_ordered_locks,
>> +   .save_context  = save_context
>>  };
>>
>>  /* Fill in scheduler API calls */
>> --
>> 2.7.4
>>

[lng-odp] [Bug 2798] ODP random and other crypto functions (possibly) not thread safe?

[bugzilla-daemon]

https://bugs.linaro.org/show_bug.cgi?id=2798

Mike Holmes  changed:

   What|Removed |Added

 CC||mike.hol...@linaro.org

--- Comment #4 from Mike Holmes  ---
Soak test still shows a problem.
Suggest we generate a specific test for odp_random_data() in thread mode.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[lng-odp] [API-NEXT PATCH] api: ipsec: packet transformation follows RFCs

[Petri Savolainen]

Add explicit requirement that IPSEC in-/outbound operations
transform packet headers according to the standards (RFCs).

Signed-off-by: Petri Savolainen 
---
 include/odp/api/spec/ipsec.h | 16 +---
 1 file changed, 13 insertions(+), 3 deletions(-)

diff --git a/include/odp/api/spec/ipsec.h b/include/odp/api/spec/ipsec.h
index 255bb79..66222d8 100644
--- a/include/odp/api/spec/ipsec.h
+++ b/include/odp/api/spec/ipsec.h
@@ -726,9 +726,6 @@ typedef struct odp_ipsec_op_result_t {
 * @see odp_packet_l3_offset(), odp_packet_l4_offset(),
 *  odp_packet_has_ipv4(), odp_packet_has_ipv6(),
 *  odp_packet_has_ipfrag(), odp_packet_has_ipsec()
-*
-* @note The amount and content of packet data before the IP header is
-*   implementation specific.
 */
odp_packet_t *pkt;
 
@@ -770,6 +767,14 @@ typedef struct odp_ipsec_op_result_t {
  * input 'pkt' array to output 'pkt' array. Packet order is not guaranteed
  * between calling threads.
  *
+ * Input packets must not be IP fragments.
+ *
+ * The operation does packet transformation according to IPSEC standards (see
+ * e.g. RFC 4302 and 4303). Resulting packets are well formed, reconstructed
+ * original IP packets, with IPSEC headers removed and valid header field 
values
+ * restored. The amount and content of packet data before the IP header is
+ * undefined.
+ *
  * @param input   Operation input parameters
  * @param[out]output  Operation results
  *
@@ -804,6 +809,11 @@ int odp_ipsec_in(const odp_ipsec_op_param_t *input,
  * input 'pkt' array to output 'pkt' array. Packet order is not guaranteed
  * between calling threads.
  *
+ * The operation does packet transformation according to IPSEC standards (see
+ * e.g. RFC 4302 and 4303). Resulting packets are well formed IP packets
+ * with IPSEC, etc headers constructed according to the standards. The amount
+ * and content of packet data before the IP header is undefined.
+ *
  * @param input   Operation input parameters
  * @param[out]output  Operation results
  *
-- 
2.8.1

[lng-odp] [Bug 2782] LCOV: _odp_atomic_u64_cmp_xchg_strong_mm reported not tested

[bugzilla-daemon]

https://bugs.linaro.org/show_bug.cgi?id=2782

Mike Holmes  changed:

   What|Removed |Added

 Resolution|--- |WONTFIX
 Status|CONFIRMED   |RESOLVED

--- Comment #2 from Mike Holmes  ---
Only applicable on old HW

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[lng-odp] Next -> master point release differences

[Mike Holmes]

http://people.linaro.org/~mike.holmes/diff-abi/libodp-compat_report.html

-- 
Mike Holmes
Program Manager - Linaro Networking Group
Linaro.org  *│ *Open source software for ARM SoCs
"Work should be fun and collaborative, the rest follows"

Re: [lng-odp] [API-NEXT PATCH] api: ipsec: packet transformation follows RFCs

[Bill Fischofer]

On Thu, Jan 12, 2017 at 8:45 AM, Petri Savolainen
 wrote:
> Add explicit requirement that IPSEC in-/outbound operations
> transform packet headers according to the standards (RFCs).
>
> Signed-off-by: Petri Savolainen 

Reviewed-by: Bill Fischofer 

> ---
>  include/odp/api/spec/ipsec.h | 16 +---
>  1 file changed, 13 insertions(+), 3 deletions(-)
>
> diff --git a/include/odp/api/spec/ipsec.h b/include/odp/api/spec/ipsec.h
> index 255bb79..66222d8 100644
> --- a/include/odp/api/spec/ipsec.h
> +++ b/include/odp/api/spec/ipsec.h
> @@ -726,9 +726,6 @@ typedef struct odp_ipsec_op_result_t {
>  * @see odp_packet_l3_offset(), odp_packet_l4_offset(),
>  *  odp_packet_has_ipv4(), odp_packet_has_ipv6(),
>  *  odp_packet_has_ipfrag(), odp_packet_has_ipsec()
> -*
> -* @note The amount and content of packet data before the IP header is
> -*   implementation specific.
>  */
> odp_packet_t *pkt;
>
> @@ -770,6 +767,14 @@ typedef struct odp_ipsec_op_result_t {
>   * input 'pkt' array to output 'pkt' array. Packet order is not guaranteed
>   * between calling threads.
>   *
> + * Input packets must not be IP fragments.
> + *
> + * The operation does packet transformation according to IPSEC standards (see
> + * e.g. RFC 4302 and 4303). Resulting packets are well formed, reconstructed
> + * original IP packets, with IPSEC headers removed and valid header field 
> values
> + * restored. The amount and content of packet data before the IP header is
> + * undefined.
> + *
>   * @param input   Operation input parameters
>   * @param[out]output  Operation results
>   *
> @@ -804,6 +809,11 @@ int odp_ipsec_in(const odp_ipsec_op_param_t *input,
>   * input 'pkt' array to output 'pkt' array. Packet order is not guaranteed
>   * between calling threads.
>   *
> + * The operation does packet transformation according to IPSEC standards (see
> + * e.g. RFC 4302 and 4303). Resulting packets are well formed IP packets
> + * with IPSEC, etc headers constructed according to the standards. The amount
> + * and content of packet data before the IP header is undefined.
> + *
>   * @param input   Operation input parameters
>   * @param[out]output  Operation results
>   *
> --
> 2.8.1
>

[lng-odp] [Bug 2816] New: Packet reference performance tuning needed

[bugzilla-daemon]

https://bugs.linaro.org/show_bug.cgi?id=2816

Bug ID: 2816
   Summary: Packet reference performance tuning needed
   Product: OpenDataPlane - linux- generic reference
   Version: api-next
  Hardware: Other
OS: Linux
Status: UNCONFIRMED
  Severity: enhancement
  Priority: ---
 Component: Buffers & Packets
  Assignee: bill.fischo...@linaro.org
  Reporter: bill.fischo...@linaro.org
CC: lng-odp@lists.linaro.org
  Target Milestone: ---

As reported by Petri, for alloc/free intensive programs that do not use
references, the initial packet reference code introduces significant extra
pathlength. This needs to be tuned to eliminate the excess overhead.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[lng-odp] [Bug 2816] Packet reference performance tuning needed

[bugzilla-daemon]

https://bugs.linaro.org/show_bug.cgi?id=2816

--- Comment #1 from Bill Fischofer  ---
Initial problem report from Petri's measurements:

This patch is now merged, although I had some doubts that it has bad impact on
performance. Here are some performance results for couple of simple, single
thread packet alloc/free test cases. No references involved, just plain packets
as before.

Test results before and after "linux-generic: packet: implement reference
apis":

CPU cycles per operation
before  after
packet_alloc127.8   250.9   +96 %
packet_alloc_multi  873.7   1538.8  +76 %
packet_free 31  116.8   +277 %
packet_free_multi   214.5   1369.2  +538 %
packet_alloc_free   73.4193.7   +164 %
packet_alloc_free_multi 286.1   1228.8  +330 %


Huge performance degradation. Numbers are now many times worse than before or
after my optimizations. To me this shows that almost a complete rewrite (or
revert) is needed.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

[lng-odp] [API-NEXT PATCH] linux-generic: packet: tune alloc/free performance for non-reference paths

[Bill Fischofer]

As reported in Bug https://bugs.linaro.org/show_bug.cgi?id=2816 packet
alloc/free performance for non-references appears significantly degraded
after the introduction of packet reference support. Add fastpaths for
non-reference packet frees to optimize this.

Signed-off-by: Bill Fischofer 
---
 platform/linux-generic/odp_packet.c | 15 ---
 1 file changed, 12 insertions(+), 3 deletions(-)

diff --git a/platform/linux-generic/odp_packet.c 
b/platform/linux-generic/odp_packet.c
index 170965a..fa2303b 100644
--- a/platform/linux-generic/odp_packet.c
+++ b/platform/linux-generic/odp_packet.c
@@ -603,14 +603,23 @@ static inline void packet_free(odp_packet_hdr_t *pkt_hdr)
 {
odp_packet_hdr_t *ref_hdr;
uint32_t ref_count;
+   int num_seg;
 
do {
ref_hdr = pkt_hdr->ref_hdr;
ref_count = packet_ref_count(pkt_hdr) - 1;
-   free_bufs(pkt_hdr, 0, pkt_hdr->buf_hdr.segcount);
+   num_seg = pkt_hdr->buf_hdr.segcount;
 
-   if (ref_count == 1)
-   pkt_hdr->unshared_len = pkt_hdr->frame_len;
+   if (odp_likely((CONFIG_PACKET_MAX_SEGS == 1 || num_seg == 1) &&
+  ref_count == 0)) {
+   buffer_free_multi((odp_buffer_t *)
+ &pkt_hdr->buf_hdr.handle.handle, 1);
+   } else {
+   free_bufs(pkt_hdr, 0, num_seg);
+
+   if (ref_count == 1)
+   pkt_hdr->unshared_len = pkt_hdr->frame_len;
+   }
 
pkt_hdr = ref_hdr;
} while (pkt_hdr);
-- 
2.9.3

[lng-odp] [Bug 2816] Packet reference performance tuning needed

[bugzilla-daemon]

https://bugs.linaro.org/show_bug.cgi?id=2816

Bill Fischofer  changed:

   What|Removed |Added

 Status|UNCONFIRMED |IN_PROGRESS
 Ever confirmed|0   |1

--- Comment #2 from Bill Fischofer  ---
Patch http://patches.opendataplane.org/patch/7857/ posted as initial step in
resolving this issue.

-- 
You are receiving this mail because:
You are on the CC list for the bug.

Re: [lng-odp] [API-NEXT PATCHv6 2/5] linux-generic: packet: implement reference apis

[Bill Fischofer]

I've posted patch http://patches.opendataplane.org/patch/7857/ as an
initial step towards resolving this issue. Petri: can you make your
test program available to me so I can try to reproduce your results
locally. I don't have quite the scale or controlled environment that
you do but this should save iteration steps.

This patch adds a fastpath to odp_packet_free() that should more
closely match the pre-refs code for non-reference packets.
odp_packet_free_multi() is difficult because there is no guarantee
that all of the packets passed to this are non-references so I'm not
sure how to match the prior code exactly in that case. This code just
calls packet_free() for each of the buffers so these individual calls
will be optimized but this still done via individual calls rather than
a single batched call.

On the alloc side I'm not sure what could be going on other than the
fact that to support references requires additional packet metadata
which probably spills onto an extra cache line compared to the
original code. As noted earlier, the alloc() changes are simply
initializing this data, which shouldn't be more than a few extra
cycles.

Thanks.

On Thu, Jan 12, 2017 at 7:22 AM, Bill Fischofer
 wrote:
> On Thu, Jan 12, 2017 at 6:58 AM, Bill Fischofer
>  wrote:
>> On Thu, Jan 12, 2017 at 6:52 AM, Savolainen, Petri (Nokia - FI/Espoo)
>>  wrote:
>>>
 >> > Huge performance degradation. Numbers are now many times worse than
 >> before or after my optimizations. To me this shows that almost a
 complete
 >> rewrite (or revert) is needed.
 >>
 >> My guess is this is due to the atomics needed for reference counting
 >> not being properly inlined internally. I know you did similar
 >> optimizations for ticketlocks. Let me look into this and post a patch
 >> that does the same for the atomics.
 >
 > Atomics are inlined already. Also atomic operations should not be
 required if there's a single refence.

 The ref_count still needs to be initialized on alloc and decremented
 on free to see if the packet should really be freed. The only way to
 know whether you're dealing with a single reference or not is to
 check. Do you have a breakdown of where the hotspots are? These
 numbers do seem high.

>>>
>>> No, I didn't look into it any deeper.
>>
>> If we look at the comparative pathlength for packet_alloc(), the only
>> changes are to the internal routines packet_init() and
>> init_segments(). The rest of the code is identical.
>>
>> For packet_init() the delta is these added two lines:
>>
>> /* By default packet has no references */
>> pkt_hdr->unshared_len = len;
>> pkt_hdr->ref_hdr = NULL;
>>
>> For init_segments() the delta is similarly tiny:
>>
>> packet_ref_count_set(hdr, 1);
>>
>> Where this is an inline function that expands to
>> odp_atomic_init_u32(&hdr->ref_count, 1);
>>
>> The changes on the packet_free() side are similarly tiny. Basically we
>> decrement the ref_count and only actually free the buffer if the
>> ref_count is now zero.
>>
>> I don't see how these translate into a near doubling of cycles for
>> these tests. It seems there must be something else going on.
>
> I think I see some potential larger deltas in the packet free path.
> I'll dig into this further and post a patch .
>
>>
>>>
>>> -Petri

Re: [lng-odp] master release status

[Bill Fischofer]

On Thu, Jan 12, 2017 at 2:43 AM, Savolainen, Petri (Nokia - FI/Espoo)
 wrote:
> Hi,
>
> What's the status of upgrading master with latest api-next commits and 
> tagging it? I can see that next branch is now at ...
>
> commit 23e7745272bd405483da737824af25e2e18c8b21
> Author: Bill Fischofer 
> Date:   Tue Jan 10 09:59:40 2017 -0600
>
> linux-generic: pool: defer ring allocation until pool creation
>
>
> ... which was considered to be needed for release. I think we should merge 
> also the next 5 commits from api-next  since those are not API changes. So, 
> this should be the tip of next / new release.
>
>
> commit da32d187cee59c0cb0c5ddf4fb0b22d387e8d951
> Author: Petri Savolainen 
> Date:   Tue Jan 10 11:19:09 2017 +0200
>
> validation: packet: limit number of failed asserts
>
>
> Packet references can be left for the next release. So, the diff between 
> master and api-next would be only the  addition of new ipsec and packet ref 
> APIs (everything else would be the same).
>

I concur with Petri on deferring the packet reference code to the next
release after this one. This represents a significant functional
change for other implementations and I'd prefer not to see their
ability to keep up with master hampered by that. We're also doing some
fine-tuning of the performance aspects of the odp-linux implementation
that can use a bit more time to mature.

>
> -Petri
>
>

Re: [lng-odp] [API-NEXT PATCH 1/2] linux-gen: ipsec: add capability function

[Bill Fischofer]

For this series:

Reviewed-and-tested-by: Bill Fischofer 

On Thu, Jan 12, 2017 at 7:21 AM, Matias Elo  wrote:
> Add IPsec capability function (groundwork for a proper IPsec
> implementation).
>
> Signed-off-by: Matias Elo 
> ---
>  platform/linux-generic/Makefile.am |  1 +
>  platform/linux-generic/odp_ipsec.c | 16 
>  2 files changed, 17 insertions(+)
>  create mode 100644 platform/linux-generic/odp_ipsec.c
>
> diff --git a/platform/linux-generic/Makefile.am 
> b/platform/linux-generic/Makefile.am
> index 999a7f5..f6c51a3 100644
> --- a/platform/linux-generic/Makefile.am
> +++ b/platform/linux-generic/Makefile.am
> @@ -185,6 +185,7 @@ __LIB__libodp_linux_la_SOURCES = \
>odp_hash.c \
>odp_init.c \
>odp_impl.c \
> +  odp_ipsec.c \
>odp_name_table.c \
>odp_packet.c \
>odp_packet_flags.c \
> diff --git a/platform/linux-generic/odp_ipsec.c 
> b/platform/linux-generic/odp_ipsec.c
> new file mode 100644
> index 000..ce7fded
> --- /dev/null
> +++ b/platform/linux-generic/odp_ipsec.c
> @@ -0,0 +1,16 @@
> +/* Copyright (c) 2017, Linaro Limited
> + * All rights reserved.
> + *
> + * SPDX-License-Identifier: BSD-3-Clause
> + */
> +
> +#include 
> +
> +#include 
> +
> +int odp_ipsec_capability(odp_ipsec_capability_t *capa)
> +{
> +   memset(capa, 0, sizeof(odp_ipsec_capability_t));
> +
> +   return 0;
> +}
> --
> 2.7.4
>

Re: [lng-odp] [API-NEXT PATCH] api: ipsec: packet transformation follows RFCs

[Nikhil Agarwal]

Reviewed-by: Nikhil Agarwal 

-Original Message-
From: lng-odp [mailto:lng-odp-boun...@lists.linaro.org] On Behalf Of Petri 
Savolainen
Sent: Thursday, January 12, 2017 8:16 PM
To: lng-odp@lists.linaro.org
Subject: [lng-odp] [API-NEXT PATCH] api: ipsec: packet transformation follows 
RFCs

Add explicit requirement that IPSEC in-/outbound operations transform packet 
headers according to the standards (RFCs).

Signed-off-by: Petri Savolainen 
---
 include/odp/api/spec/ipsec.h | 16 +---
 1 file changed, 13 insertions(+), 3 deletions(-)

diff --git a/include/odp/api/spec/ipsec.h b/include/odp/api/spec/ipsec.h index 
255bb79..66222d8 100644
--- a/include/odp/api/spec/ipsec.h
+++ b/include/odp/api/spec/ipsec.h
@@ -726,9 +726,6 @@ typedef struct odp_ipsec_op_result_t {
 * @see odp_packet_l3_offset(), odp_packet_l4_offset(),
 *  odp_packet_has_ipv4(), odp_packet_has_ipv6(),
 *  odp_packet_has_ipfrag(), odp_packet_has_ipsec()
-*
-* @note The amount and content of packet data before the IP header is
-*   implementation specific.
 */
odp_packet_t *pkt;
 
@@ -770,6 +767,14 @@ typedef struct odp_ipsec_op_result_t {
  * input 'pkt' array to output 'pkt' array. Packet order is not guaranteed
  * between calling threads.
  *
+ * Input packets must not be IP fragments.
+ *
+ * The operation does packet transformation according to IPSEC 
+ standards (see
+ * e.g. RFC 4302 and 4303). Resulting packets are well formed, 
+ reconstructed
+ * original IP packets, with IPSEC headers removed and valid header 
+ field values
+ * restored. The amount and content of packet data before the IP header 
+ is
+ * undefined.
+ *
  * @param input   Operation input parameters
  * @param[out]output  Operation results
  *
@@ -804,6 +809,11 @@ int odp_ipsec_in(const odp_ipsec_op_param_t *input,
  * input 'pkt' array to output 'pkt' array. Packet order is not guaranteed
  * between calling threads.
  *
+ * The operation does packet transformation according to IPSEC 
+ standards (see
+ * e.g. RFC 4302 and 4303). Resulting packets are well formed IP 
+ packets
+ * with IPSEC, etc headers constructed according to the standards. The 
+ amount
+ * and content of packet data before the IP header is undefined.
+ *
  * @param input   Operation input parameters
  * @param[out]output  Operation results
  *
--
2.8.1

Re: [lng-odp] [API-NEXT PATCH 4/4] validation: crypto: check auth options support before running tests

[Nikhil Agarwal]

For the series:

Reviewed-by: Nikhil Agarwal 

-Original Message-
From: lng-odp [mailto:lng-odp-boun...@lists.linaro.org] On Behalf Of Matias Elo
Sent: Tuesday, January 10, 2017 5:39 PM
To: lng-odp@lists.linaro.org
Subject: [lng-odp] [API-NEXT PATCH 4/4] validation: crypto: check auth options 
support before running tests

Skip running test if authentication options are not supported.

Signed-off-by: Matias Elo 
---
 .../validation/api/crypto/odp_crypto_test_inp.c| 72 ++
 1 file changed, 72 insertions(+)

diff --git a/test/common_plat/validation/api/crypto/odp_crypto_test_inp.c 
b/test/common_plat/validation/api/crypto/odp_crypto_test_inp.c
index 426f842..43ddb2f 100644
--- a/test/common_plat/validation/api/crypto/odp_crypto_test_inp.c
+++ b/test/common_plat/validation/api/crypto/odp_crypto_test_inp.c
@@ -21,6 +21,22 @@ struct suite_context_s {
 
 static struct suite_context_s suite_context;
 
+static const char *auth_alg_name(odp_auth_alg_t auth) {
+   switch (auth) {
+   case ODP_AUTH_ALG_NULL:
+   return "ODP_AUTH_ALG_NULL";
+   case ODP_AUTH_ALG_MD5_HMAC:
+   return "ODP_AUTH_ALG_MD5_HMAC";
+   case ODP_AUTH_ALG_SHA256_HMAC:
+   return "ODP_AUTH_ALG_SHA256_HMAC";
+   case ODP_AUTH_ALG_AES_GCM:
+   return "ODP_AUTH_ALG_AES_GCM";
+   default:
+   return "Unknown";
+   }
+}
+
 static const char *cipher_alg_name(odp_cipher_alg_t cipher)  {
switch (cipher) {
@@ -365,6 +381,41 @@ static int check_cipher_options(odp_cipher_alg_t cipher, 
uint32_t key_len,
return 1;
 }
 
+/**
+ * Check if given authentication options are supported
+ *
+ * @param authAuthentication algorithm
+ * @param key_len Key length
+ * @param digest_len  Digest length
+ *
+ * @retval non-zero if both authentication options are supported
+ * @retval 0 if both options are not supported  */ static int 
+check_auth_options(odp_auth_alg_t auth, uint32_t key_len,
+ uint32_t digest_len)
+{
+   int i;
+   int num;
+   odp_crypto_auth_capability_t capa[MAX_ALG_CAPA];
+
+   num = odp_crypto_auth_capability(auth, capa, MAX_ALG_CAPA);
+   CU_ASSERT_FATAL(num >= 1);
+
+   for (i = 0; i < num; i++) {
+   if (key_len == capa[i].key_len &&
+   digest_len == capa[i].digest_len)
+   break;
+   }
+
+   if (i == num) {
+   printf("\nUnsupported: alg=%s, key_len=%" PRIu32 ", "
+  "digest_len=%" PRIu32 "\n", auth_alg_name(auth), key_len,
+  digest_len);
+   return 0;
+   }
+   return 1;
+}
+
 static int check_alg_3des_cbc(void)
 {
return check_alg_support(ODP_CIPHER_ALG_3DES_CBC, ODP_AUTH_ALG_NULL); 
@@ -546,6 +597,9 @@ void crypto_test_enc_alg_aes128_gcm(void)
if (!check_cipher_options(ODP_CIPHER_ALG_AES_GCM,
  cipher_key.length, iv.length))
continue;
+   if (!check_auth_options(ODP_AUTH_ALG_AES_GCM,
+   auth_key.length, AES128_GCM_CHECK_LEN))
+   continue;
 
alg_test(ODP_CRYPTO_OP_ENCODE,
 ODP_CIPHER_ALG_AES_GCM,
@@ -586,6 +640,9 @@ void crypto_test_enc_alg_aes128_gcm_ovr_iv(void)
if (!check_cipher_options(ODP_CIPHER_ALG_AES_GCM,
  cipher_key.length, iv.length))
continue;
+   if (!check_auth_options(ODP_AUTH_ALG_AES_GCM,
+   auth_key.length, AES128_GCM_CHECK_LEN))
+   continue;
 
alg_test(ODP_CRYPTO_OP_ENCODE,
 ODP_CIPHER_ALG_AES_GCM,
@@ -629,6 +686,9 @@ void crypto_test_dec_alg_aes128_gcm(void)
if (!check_cipher_options(ODP_CIPHER_ALG_AES_GCM,
  cipher_key.length, iv.length))
continue;
+   if (!check_auth_options(ODP_AUTH_ALG_AES_GCM,
+   auth_key.length, AES128_GCM_CHECK_LEN))
+   continue;
 
alg_test(ODP_CRYPTO_OP_DECODE,
 ODP_CIPHER_ALG_AES_GCM,
@@ -670,6 +730,9 @@ void crypto_test_dec_alg_aes128_gcm_ovr_iv(void)
if (!check_cipher_options(ODP_CIPHER_ALG_AES_GCM,
  cipher_key.length, iv.length))
continue;
+   if (!check_auth_options(ODP_AUTH_ALG_AES_GCM,
+   auth_key.length, AES128_GCM_CHECK_LEN))
+   continue;
 
alg_test(ODP_CRYPTO_OP_DECODE,
 ODP_CIPHER_ALG_AES_GCM,
@@ -870,6 +933,10 @@ void crypto_test_alg_hmac_md5(void)
auth_key.data = hmac_md5_reference_key[i];
  

Re: [lng-odp] [API-NEXT PATCH 1/4] validation: crypto: fix hw cipher/auth algorithm check

[Nikhil Agarwal]

Can we add sync/Async in capabilities and add similar check before running 
synchronous and asynchronous suites?

Regards
Nikhil

-Original Message-
From: lng-odp [mailto:lng-odp-boun...@lists.linaro.org] On Behalf Of Matias Elo
Sent: Tuesday, January 10, 2017 5:39 PM
To: lng-odp@lists.linaro.org
Subject: [lng-odp] [API-NEXT PATCH 1/4] validation: crypto: fix hw cipher/auth 
algorithm check

Some algorithms may be implemented using hardware and some using software.
All supported algorithms should be set in capacity.auths /capacity.ciphers 
independent of implementation types.

Signed-off-by: Matias Elo 
---
 .../validation/api/crypto/odp_crypto_test_inp.c| 69 ++
 1 file changed, 31 insertions(+), 38 deletions(-)

diff --git a/test/common_plat/validation/api/crypto/odp_crypto_test_inp.c 
b/test/common_plat/validation/api/crypto/odp_crypto_test_inp.c
index de9d6e4..db58344 100644
--- a/test/common_plat/validation/api/crypto/odp_crypto_test_inp.c
+++ b/test/common_plat/validation/api/crypto/odp_crypto_test_inp.c
@@ -47,7 +47,7 @@ static void alg_test(odp_crypto_op_t op,
 uint32_t digest_len)
 {
odp_crypto_session_t session;
-   odp_crypto_capability_t capability;
+   odp_crypto_capability_t capa;
int rc;
odp_crypto_ses_create_err_t status;
odp_bool_t posted;
@@ -63,50 +63,43 @@ static void alg_test(odp_crypto_op_t op,
int num, i;
int found;
 
-   rc = odp_crypto_capability(&capability);
+   rc = odp_crypto_capability(&capa);
CU_ASSERT(!rc);
 
-   if (capability.hw_ciphers.all_bits) {
-   if (cipher_alg == ODP_CIPHER_ALG_3DES_CBC &&
-   !(capability.hw_ciphers.bit.trides_cbc))
-   rc = -1;
-   if (cipher_alg == ODP_CIPHER_ALG_AES_CBC &&
-   !(capability.hw_ciphers.bit.aes_cbc))
-   rc = -1;
-   if (cipher_alg == ODP_CIPHER_ALG_AES_GCM &&
-   !(capability.hw_ciphers.bit.aes_gcm))
-   rc = -1;
-   } else {
-   if (cipher_alg == ODP_CIPHER_ALG_3DES_CBC &&
-   !(capability.ciphers.bit.trides_cbc))
-   rc = -1;
-   if (cipher_alg == ODP_CIPHER_ALG_AES_CBC &&
-   !(capability.ciphers.bit.aes_cbc))
-   rc = -1;
-   if (cipher_alg == ODP_CIPHER_ALG_AES_GCM &&
-   !(capability.ciphers.bit.aes_gcm))
-   rc = -1;
-   }
+   if (cipher_alg == ODP_CIPHER_ALG_3DES_CBC &&
+   !(capa.ciphers.bit.trides_cbc))
+   rc = -1;
+   if (cipher_alg == ODP_CIPHER_ALG_AES_CBC &&
+   !(capa.ciphers.bit.aes_cbc))
+   rc = -1;
+   if (cipher_alg == ODP_CIPHER_ALG_AES_GCM &&
+   !(capa.ciphers.bit.aes_gcm))
+   rc = -1;
+   if (cipher_alg == ODP_CIPHER_ALG_DES &&
+   !(capa.ciphers.bit.des))
+   rc = -1;
+   if (cipher_alg == ODP_CIPHER_ALG_NULL &&
+   !(capa.ciphers.bit.null))
+   rc = -1;
 
CU_ASSERT(!rc);
+   CU_ASSERT((~capa.ciphers.all_bits & capa.hw_ciphers.all_bits) == 0);
 
-   if (capability.hw_auths.all_bits) {
-   if (auth_alg == ODP_AUTH_ALG_AES_GCM &&
-   !(capability.hw_auths.bit.aes_gcm))
-   rc = -1;
-   if (auth_alg == ODP_AUTH_ALG_NULL &&
-   !(capability.hw_auths.bit.null))
-   rc = -1;
-   } else {
-   if (auth_alg == ODP_AUTH_ALG_AES_GCM &&
-   !(capability.auths.bit.aes_gcm))
-   rc = -1;
-   if (auth_alg == ODP_AUTH_ALG_NULL &&
-   !(capability.auths.bit.null))
-   rc = -1;
-   }
+   if (auth_alg == ODP_AUTH_ALG_AES_GCM &&
+   !(capa.auths.bit.aes_gcm))
+   rc = -1;
+   if (auth_alg == ODP_AUTH_ALG_MD5_HMAC &&
+   !(capa.auths.bit.md5_hmac))
+   rc = -1;
+   if (auth_alg == ODP_AUTH_ALG_NULL &&
+   !(capa.auths.bit.null))
+   rc = -1;
+   if (auth_alg == ODP_AUTH_ALG_SHA256_HMAC &&
+   !(capa.auths.bit.sha256_hmac))
+   rc = -1;
 
CU_ASSERT(!rc);
+   CU_ASSERT((~capa.auths.all_bits & capa.hw_auths.all_bits) == 0);
 
num = odp_crypto_cipher_capability(cipher_alg, cipher_capa,
   MAX_ALG_CAPA);
--
2.7.4

Re: [lng-odp] [API-NEXT PATCH] api: ipsec: packet transformation follows RFCs

[Bala Manoharan]

Reviewed-by: Balasubramanian Manoharan 

On 13 January 2017 at 11:58, Nikhil Agarwal  wrote:
> Reviewed-by: Nikhil Agarwal 
>
> -Original Message-
> From: lng-odp [mailto:lng-odp-boun...@lists.linaro.org] On Behalf Of Petri 
> Savolainen
> Sent: Thursday, January 12, 2017 8:16 PM
> To: lng-odp@lists.linaro.org
> Subject: [lng-odp] [API-NEXT PATCH] api: ipsec: packet transformation follows 
> RFCs
>
> Add explicit requirement that IPSEC in-/outbound operations transform packet 
> headers according to the standards (RFCs).
>
> Signed-off-by: Petri Savolainen 
> ---
>  include/odp/api/spec/ipsec.h | 16 +---
>  1 file changed, 13 insertions(+), 3 deletions(-)
>
> diff --git a/include/odp/api/spec/ipsec.h b/include/odp/api/spec/ipsec.h 
> index 255bb79..66222d8 100644
> --- a/include/odp/api/spec/ipsec.h
> +++ b/include/odp/api/spec/ipsec.h
> @@ -726,9 +726,6 @@ typedef struct odp_ipsec_op_result_t {
>  * @see odp_packet_l3_offset(), odp_packet_l4_offset(),
>  *  odp_packet_has_ipv4(), odp_packet_has_ipv6(),
>  *  odp_packet_has_ipfrag(), odp_packet_has_ipsec()
> -*
> -* @note The amount and content of packet data before the IP header is
> -*   implementation specific.
>  */
> odp_packet_t *pkt;
>
> @@ -770,6 +767,14 @@ typedef struct odp_ipsec_op_result_t {
>   * input 'pkt' array to output 'pkt' array. Packet order is not guaranteed
>   * between calling threads.
>   *
> + * Input packets must not be IP fragments.
> + *
> + * The operation does packet transformation according to IPSEC
> + standards (see
> + * e.g. RFC 4302 and 4303). Resulting packets are well formed,
> + reconstructed
> + * original IP packets, with IPSEC headers removed and valid header
> + field values
> + * restored. The amount and content of packet data before the IP header
> + is
> + * undefined.
> + *
>   * @param input   Operation input parameters
>   * @param[out]output  Operation results
>   *
> @@ -804,6 +809,11 @@ int odp_ipsec_in(const odp_ipsec_op_param_t *input,
>   * input 'pkt' array to output 'pkt' array. Packet order is not guaranteed
>   * between calling threads.
>   *
> + * The operation does packet transformation according to IPSEC
> + standards (see
> + * e.g. RFC 4302 and 4303). Resulting packets are well formed IP
> + packets
> + * with IPSEC, etc headers constructed according to the standards. The
> + amount
> + * and content of packet data before the IP header is undefined.
> + *
>   * @param input   Operation input parameters
>   * @param[out]output  Operation results
>   *
> --
> 2.8.1
>

[lng-odp] [API-NEXT PATCHv7 0/6] small memory amount allocator for drv shm

[Christophe Milard]

Since V6:
 -rebased

Since V5:
 -title fix (Yi)
 -rebased
 -flag name fix (Christophe)

Since V4:
 -fix for clang

Since V3:
 -copyright year changed

Since V2:
 -comment and doxygen fixes (Petri)

Since V1:
 -Common south pool (for all drivers) suppressed. (Petri)
 -function to create and destroy the pools exposed to the interface,
  so that each user create his own pools (Petri)
 -support for fixed size allocator added (Petri, Bill)
 -Creation parameter now includes the min-alloc/max_alloc (Petri)
 -Pool creation parameter passed as struct (Petri)

This patch series aims at giving the ability for units using the ODP
drv (south) interface to allocate small unit of memory (hence going lower
than the page size, which is the lower limit for odp_drvshm_reserve()).
South interface users can create and destroy pool of memory using
odpdrv_shm_pool_create() odpdrv_shm_pool_destroy().
The interface also provides two functions: odpdrv_shm_pool_alloc()
and odpdrv_shm_pool_free() whose usage is very similar to
malloc/free, but which also guarantee address uniqueness amoung all
ODP threads).
Internally, for the linux-gen implementation, this is implemented thanks
to a buddy and slab allocator handling the preallocated memory.
Enumerators typically will need this to contruct the list of enumerated
devices. ODP will also need it for building list of things such as
enumerators, devio, drivers...
(PS:I guess this concept will be needed as well on the north (API) interface
when we start looking into having things working "in process mode")

Christophe Milard (6):
  linux-gen: _ishm: adding buddy and slab allocation
  drv: adding odpdrv_shm_pool functions
  linux-gen: drv: shm: adding pool allocator
  test: drv: shm: adding basic buddy allocation tests
  test: drv: shm: adding basic fixed size allocation tests
  test: drv: shm: adding buddy allocation stress tests

 include/odp/drv/spec/shm.h |  94 +++
 platform/linux-generic/Makefile.am |   2 +
 platform/linux-generic/_ishm.c |  14 +-
 platform/linux-generic/_ishmpool.c | 811 +
 platform/linux-generic/drv_shm.c   |  44 ++
 .../linux-generic/include/_ishmpool_internal.h |  56 ++
 .../linux-generic/include/odp/drv/plat/shm_types.h |   3 +
 .../common_plat/validation/drv/drvshmem/drvshmem.c | 355 +
 .../common_plat/validation/drv/drvshmem/drvshmem.h |   3 +
 9 files changed, 1381 insertions(+), 1 deletion(-)
 create mode 100644 platform/linux-generic/_ishmpool.c
 create mode 100644 platform/linux-generic/include/_ishmpool_internal.h

-- 
2.7.4

[lng-odp] [API-NEXT PATCHv7 2/6] drv: adding odpdrv_shm_pool functions

[Christophe Milard]

Adding functions to create and destroy memory pools (from which memory
can be allocated and freed) are added.
These functions enable the usage of small memory amount (compared to
drvshm_reserve() whose granularity is the page size).
The usage of this pool guatantees that allocated memory is sharable
between ODP threads. (using malloc would not work when ODP threads
are linux processes).

Signed-off-by: Christophe Milard 
Reviewed-by: Petri Savolainen 
---
 include/odp/drv/spec/shm.h | 94 ++
 .../linux-generic/include/odp/drv/plat/shm_types.h |  3 +
 2 files changed, 97 insertions(+)

diff --git a/include/odp/drv/spec/shm.h b/include/odp/drv/spec/shm.h
index ef64f5d..64124c0 100644
--- a/include/odp/drv/spec/shm.h
+++ b/include/odp/drv/spec/shm.h
@@ -220,6 +220,100 @@ int odpdrv_shm_print_all(const char *title);
 uint64_t odpdrv_shm_to_u64(odpdrv_shm_t hdl);
 
 /**
+ * drv shm pool parameters
+ * Used to communicate pool creation options.
+ */
+typedef struct {
+   /** Sum of all (simultaneous) allocs (bytes)*/
+   uint64_t pool_size;
+
+   /** Minimum alloc size user will request from pool (bytes)*/
+   uint64_t min_alloc;
+
+   /** Maximum alloc size user will request from pool (bytes)*/
+   uint64_t max_alloc;
+} odpdrv_shm_pool_param_t;
+
+/**
+ * @typedef odpdrv_shm_pool_t
+ * odpdrv shared memory pool
+ */
+
+/**
+ * Create a memory pool
+ *
+ * This routine is used to create a memory pool. The use of pool name is
+ * optional.
+ * Unique names are not required. However, odpdrv_shm_pool_lookup()
+ * returns only a single matching pool.
+ *
+ * @param name Name of the pool or NULL.
+ * @param paramPool parameters.
+ *
+ * @return Handle of the created drv shm memory pool
+ * @retval ODPDRV_SHM_POOL_INVALID  Pool could not be created
+ */
+odpdrv_shm_pool_t odpdrv_shm_pool_create(const char *pool_name,
+odpdrv_shm_pool_param_t *param);
+
+/**
+ * Destroy a pool previously created by odpdrv_shm_pool_create()
+ *
+ * @param poolHandle of the pool to be destroyed
+ *
+ * @retval 0 Success
+ * @retval <0 Failure
+ *
+ * @note This routine destroys a previously created pool, and will destroy any
+ * internal shared memory objects associated with the pool. Results are
+ * undefined if an attempt is made to destroy a pool that contains allocated
+ * or otherwise active allocations.
+ */
+int odpdrv_shm_pool_destroy(odpdrv_shm_pool_t pool);
+
+/**
+ * Find a memory pool by name
+ *
+ * @param name  Name of the pool
+ *
+ * @return Handle of the first matching pool
+ * @retval ODPDRV_SHM_POOL_INVALID Pool could not be found
+ */
+odpdrv_shm_pool_t odpdrv_shm_pool_lookup(const char *name);
+
+/**
+ * Allocate memory from a memory pool
+ *
+ * @param pool  Memory pool handle
+ * @param size  Number of bytes to allocate (bytes)
+ *
+ * @return A pointer to the allocated memory
+ * @retval NULL on error.
+ */
+void *odpdrv_shm_pool_alloc(odpdrv_shm_pool_t pool, uint64_t size);
+
+/**
+ * Free memory  back to a memory pool
+ *
+ * @param pool  Memory pool handle
+ * @param addr  pointer to a previously allocated memory
+ * (as returned by a previous call to odpdrv_shm_pool_alloc)
+ */
+void odpdrv_shm_pool_free(odpdrv_shm_pool_t pool, void *addr);
+
+/**
+ * Print memory pool info
+ *
+ * @param title A string to be printed as a title (e.g. location)
+ * @param pool  Memory pool handle
+ *
+ * @return 0 on success, negative value if pool inconsistency is detected.
+ *
+ * @note This routine writes implementation-defined information about the
+ * specified pool to the ODP log. The intended use is for debugging.
+ */
+int  odpdrv_shm_pool_print(const char *title, odpdrv_shm_pool_t pool);
+/**
  * @}
  */
 
diff --git a/platform/linux-generic/include/odp/drv/plat/shm_types.h 
b/platform/linux-generic/include/odp/drv/plat/shm_types.h
index c48eeca..50a0837 100644
--- a/platform/linux-generic/include/odp/drv/plat/shm_types.h
+++ b/platform/linux-generic/include/odp/drv/plat/shm_types.h
@@ -35,6 +35,9 @@ static inline uint64_t odpdrv_shm_to_u64(odpdrv_shm_t hdl)
return _odpdrv_pri(hdl);
 }
 
+typedef ODPDRV_HANDLE_T(odpdrv_shm_pool_t);
+
+#define ODPDRV_SHM_POOL_INVALID _odpdrv_cast_scalar(odpdrv_shm_pool_t, NULL)
 /**
  * @}
  */
-- 
2.7.4

[lng-odp] [API-NEXT PATCHv7 3/6] linux-gen: drv: shm: adding pool allocator

[Christophe Milard]

Adding functions to create memory pools and allocate / free memory from
the created pools.
These functions calls their _ishm conterpart, of course.

Signed-off-by: Christophe Milard 
---
 platform/linux-generic/drv_shm.c | 44 
 1 file changed, 44 insertions(+)

diff --git a/platform/linux-generic/drv_shm.c b/platform/linux-generic/drv_shm.c
index 9b2560d..325632e 100644
--- a/platform/linux-generic/drv_shm.c
+++ b/platform/linux-generic/drv_shm.c
@@ -8,6 +8,7 @@
 #include 
 #include 
 #include <_ishm_internal.h>
+#include <_ishmpool_internal.h>
 
 static inline uint32_t from_handle(odpdrv_shm_t shm)
 {
@@ -100,3 +101,46 @@ int odpdrv_shm_print_all(const char *title)
 {
return _odp_ishm_status(title);
 }
+
+odpdrv_shm_pool_t odpdrv_shm_pool_create(const char *pool_name,
+odpdrv_shm_pool_param_t *param)
+{
+   int flags;
+
+   /* force unique address for all ODP threads */
+   flags = _ODP_ISHM_SINGLE_VA;
+   return (odpdrv_shm_pool_t)_odp_ishm_pool_create(pool_name,
+   param->pool_size,
+   param->min_alloc,
+   param->max_alloc,
+   flags);
+}
+
+int odpdrv_shm_pool_destroy(odpdrv_shm_pool_t pool)
+{
+   return _odp_ishm_pool_destroy((_odp_ishm_pool_t *)(void*)pool);
+}
+
+odpdrv_shm_pool_t odpdrv_shm_pool_lookup(const char *name)
+{
+   return (odpdrv_shm_pool_t)_odp_ishm_pool_lookup(name);
+}
+
+void *odpdrv_shm_pool_alloc(odpdrv_shm_pool_t pool, uint64_t size)
+{
+   return _odp_ishm_pool_alloc((_odp_ishm_pool_t *)(void*)pool, size);
+}
+
+void odpdrv_shm_pool_free(odpdrv_shm_pool_t pool, void *addr)
+{
+   (void)_odp_ishm_pool_free((_odp_ishm_pool_t *)(void*)pool, addr);
+}
+
+int odpdrv_shm_pool_print(const char *title, odpdrv_shm_pool_t pool)
+{
+   return _odp_ishm_pool_status(title, (_odp_ishm_pool_t *)(void*)pool);
+}
+
+/**
+ * @}
+ */
-- 
2.7.4

[lng-odp] [API-NEXT PATCHv7 4/6] test: drv: shm: adding basic buddy allocation tests

[Christophe Milard]

Basic tests for odpdrv_shm_pool are added here, creating a buddy
pool and performing basic alloc/free on it

Signed-off-by: Christophe Milard 
---
 .../common_plat/validation/drv/drvshmem/drvshmem.c | 92 ++
 .../common_plat/validation/drv/drvshmem/drvshmem.h |  1 +
 2 files changed, 93 insertions(+)

diff --git a/test/common_plat/validation/drv/drvshmem/drvshmem.c 
b/test/common_plat/validation/drv/drvshmem/drvshmem.c
index 0247a03..5843573 100644
--- a/test/common_plat/validation/drv/drvshmem/drvshmem.c
+++ b/test/common_plat/validation/drv/drvshmem/drvshmem.c
@@ -21,6 +21,12 @@
 #define STRESS_RANDOM_SZ 5
 #define STRESS_ITERATION 5000
 
+#define POOL_NAME "test_pool"
+#define POOL_SZ (1UL << 20)/* 1 MBytes */
+#define TEST_SZ 1000
+#define SZ_1K   1024
+#define BUFF_PATTERN 0xA3
+
 typedef enum {
STRESS_FREE, /* entry is free and can be allocated */
STRESS_BUSY, /* entry is being processed: don't touch */
@@ -762,11 +768,97 @@ void drvshmem_test_stress(void)
CU_ASSERT(odpdrv_shm_print_all("After stress tests") == base);
 }
 
+void drvshmem_test_buddy_basic(void)
+{
+   odpdrv_shm_pool_param_t pool_params;
+   odpdrv_shm_pool_t pool, found_pool;
+   uint8_t *buff;
+   uint8_t *addrs[TEST_SZ];
+   uint8_t length;
+   int i, j;
+
+   /* create a pool and check that it can be looked up */
+   pool_params.pool_size = POOL_SZ;
+   pool_params.min_alloc = 1;
+   pool_params.max_alloc = POOL_SZ;
+   pool = odpdrv_shm_pool_create(POOL_NAME, &pool_params);
+   found_pool = odpdrv_shm_pool_lookup(POOL_NAME);
+   CU_ASSERT(found_pool == pool);
+
+   /* alloc a 1k buffer, filling its contents: */
+   buff = odpdrv_shm_pool_alloc(pool, SZ_1K);
+   CU_ASSERT_PTR_NOT_NULL(buff);
+   for (i = 0; i < SZ_1K; i++)
+   buff[i] = BUFF_PATTERN;
+   odpdrv_shm_pool_print("buddy test: 1K reserved", pool);
+
+   /* alloc as many buffer a possible on increseasing sz */
+   for (i = 0; i < TEST_SZ; i++) {
+   length = i * 16;
+   addrs[i] = odpdrv_shm_pool_alloc(pool, length);
+   /* if alloc was success, fill buffer for later check */
+   if (addrs[i]) {
+   for (j = 0; j < length; j++)
+   addrs[i][j] = (uint8_t)(length & 0xFF);
+   }
+   }
+   odpdrv_shm_pool_print("buddy test: after many mallocs", pool);
+
+   /* release every 3rth buffer, checking contents: */
+   for (i = 0; i < TEST_SZ; i += 3) {
+   /* if buffer was allocated, check the pattern in it */
+   if (addrs[i]) {
+   length = i * 16;
+   for (j = 0; j < length; j++)
+   CU_ASSERT(addrs[i][j] ==
+ (uint8_t)(length & 0xFF));
+   }
+   odpdrv_shm_pool_free(pool, addrs[i]);
+   }
+   odpdrv_shm_pool_print("buddy test: after 1/3 free:", pool);
+
+   /* realloc them:*/
+   for (i = 0; i < TEST_SZ; i += 3) {
+   length = i * 16;
+   addrs[i] = odpdrv_shm_pool_alloc(pool, length);
+   /* if alloc was success, fill buffer for later check */
+   if (addrs[i]) {
+   for (j = 0; j < length; j++)
+   addrs[i][j] = (uint8_t)(length & 0xFF);
+   }
+   }
+   odpdrv_shm_pool_print("buddy test: after realloc:", pool);
+
+   /* free all (except buff), checking contents: */
+   for (i = 0; i < TEST_SZ; i++) {
+   /* if buffer was allocated, check the pattern in it */
+   if (addrs[i]) {
+   length = i * 16;
+   for (j = 0; j < length; j++)
+   CU_ASSERT(addrs[i][j] ==
+ (uint8_t)(length & 0xFF))
+   }
+   odpdrv_shm_pool_free(pool, addrs[i]);
+   }
+   odpdrv_shm_pool_print("buddy test: after all but 1K free:", pool);
+
+   /* check contents of our initial 1K buffer: */
+   for (i = 0; i < SZ_1K; i++)
+   CU_ASSERT((buff[i] == BUFF_PATTERN))
+   odpdrv_shm_pool_free(pool, buff);
+
+   odpdrv_shm_pool_print("buddy test: after all free", pool);
+
+   /* destroy pool: */
+   odpdrv_shm_pool_destroy(pool);
+}
+
 odp_testinfo_t drvshmem_suite[] = {
ODP_TEST_INFO(drvshmem_test_basic),
ODP_TEST_INFO(drvshmem_test_reserve_after_fork),
ODP_TEST_INFO(drvshmem_test_singleva_after_fork),
ODP_TEST_INFO(drvshmem_test_stress),
+   ODP_TEST_INFO(drvshmem_test_buddy_basic),
ODP_TEST_INFO_NULL,
 };
 
diff --git a/test/common_plat/validation/drv/drvshmem/drvshmem.h 
b/test/common_plat/validation/drv/drvshmem/drvshmem.h
index f4c26a1..ab45f7c 100644
--- a/test/common_plat/validation/drv/drvshmem/drvshmem.h
+++ b/t

[lng-odp] [API-NEXT PATCHv7 5/6] test: drv: shm: adding basic fixed size allocation tests

[Christophe Milard]

Basic tests for odpdrv_shm_pool are added here, creating a fixed size
pool and performing basic alloc/free on it

Signed-off-by: Christophe Milard 
---
 .../common_plat/validation/drv/drvshmem/drvshmem.c | 86 ++
 .../common_plat/validation/drv/drvshmem/drvshmem.h |  1 +
 2 files changed, 87 insertions(+)

diff --git a/test/common_plat/validation/drv/drvshmem/drvshmem.c 
b/test/common_plat/validation/drv/drvshmem/drvshmem.c
index 5843573..d4dedea 100644
--- a/test/common_plat/validation/drv/drvshmem/drvshmem.c
+++ b/test/common_plat/validation/drv/drvshmem/drvshmem.c
@@ -853,12 +853,98 @@ void drvshmem_test_buddy_basic(void)
odpdrv_shm_pool_destroy(pool);
 }
 
+void drvshmem_test_slab_basic(void)
+{
+   odpdrv_shm_pool_param_t pool_params;
+   odpdrv_shm_pool_t pool, found_pool;
+   uint8_t *buff;
+   uint8_t *addrs[TEST_SZ];
+   uint16_t length;
+   int i, j;
+
+   /* create a pool and check that it can be looked up */
+   pool_params.pool_size = POOL_SZ;
+   pool_params.min_alloc = SZ_1K; /* constant size will give slab */
+   pool_params.max_alloc = SZ_1K;
+   pool = odpdrv_shm_pool_create(POOL_NAME, &pool_params);
+   found_pool = odpdrv_shm_pool_lookup(POOL_NAME);
+   CU_ASSERT(found_pool == pool);
+
+   /* alloc a 1k buffer, filling its contents: */
+   buff = odpdrv_shm_pool_alloc(pool, SZ_1K);
+   CU_ASSERT_PTR_NOT_NULL(buff);
+   for (i = 0; i < SZ_1K; i++)
+   buff[i] = BUFF_PATTERN;
+   odpdrv_shm_pool_print("buddy test: 1K reserved", pool);
+
+   /* alloc as many 1K buffer a possible */
+   for (i = 0; i < TEST_SZ; i++) {
+   length = SZ_1K;
+   addrs[i] = odpdrv_shm_pool_alloc(pool, length);
+   /* if alloc was success, fill buffer for later check */
+   if (addrs[i]) {
+   for (j = 0; j < length; j++)
+   addrs[i][j] = (uint8_t)(length & 0xFF);
+   }
+   }
+   odpdrv_shm_pool_print("slab test: after many mallocs", pool);
+
+   /* release every 3rth buffer, checking contents: */
+   for (i = 0; i < TEST_SZ; i += 3) {
+   /* if buffer was allocated, check the pattern in it */
+   if (addrs[i]) {
+   length = SZ_1K;
+   for (j = 0; j < length; j++)
+   CU_ASSERT(addrs[i][j] ==
+ (uint8_t)(length & 0xFF));
+   }
+   odpdrv_shm_pool_free(pool, addrs[i]);
+   }
+   odpdrv_shm_pool_print("slab test: after 1/3 free:", pool);
+
+   /* realloc them:*/
+   for (i = 0; i < TEST_SZ; i += 3) {
+   length = SZ_1K;
+   addrs[i] = odpdrv_shm_pool_alloc(pool, length);
+   /* if alloc was success, fill buffer for later check */
+   if (addrs[i]) {
+   for (j = 0; j < length; j++)
+   addrs[i][j] = (uint8_t)(length & 0xFF);
+   }
+   }
+   odpdrv_shm_pool_print("slab test: after realloc:", pool);
+
+   /* free all (except buff), checking contents: */
+   for (i = 0; i < TEST_SZ; i++) {
+   /* if buffer was allocated, check the pattern in it */
+   if (addrs[i]) {
+   length = SZ_1K;
+   for (j = 0; j < length; j++)
+   CU_ASSERT(addrs[i][j] ==
+ (uint8_t)(length & 0xFF))
+   }
+   odpdrv_shm_pool_free(pool, addrs[i]);
+   }
+   odpdrv_shm_pool_print("slab test: after all but 1K free:", pool);
+
+   /* check contents of our initial 1K buffer: */
+   for (i = 0; i < SZ_1K; i++)
+   CU_ASSERT((buff[i] == BUFF_PATTERN))
+   odpdrv_shm_pool_free(pool, buff);
+
+   odpdrv_shm_pool_print("slab test: after all free", pool);
+
+   /* destroy pool: */
+   odpdrv_shm_pool_destroy(pool);
+}
+
 odp_testinfo_t drvshmem_suite[] = {
ODP_TEST_INFO(drvshmem_test_basic),
ODP_TEST_INFO(drvshmem_test_reserve_after_fork),
ODP_TEST_INFO(drvshmem_test_singleva_after_fork),
ODP_TEST_INFO(drvshmem_test_stress),
ODP_TEST_INFO(drvshmem_test_buddy_basic),
+   ODP_TEST_INFO(drvshmem_test_slab_basic),
ODP_TEST_INFO_NULL,
 };
 
diff --git a/test/common_plat/validation/drv/drvshmem/drvshmem.h 
b/test/common_plat/validation/drv/drvshmem/drvshmem.h
index ab45f7c..fdc1080 100644
--- a/test/common_plat/validation/drv/drvshmem/drvshmem.h
+++ b/test/common_plat/validation/drv/drvshmem/drvshmem.h
@@ -15,6 +15,7 @@ void drvshmem_test_reserve_after_fork(void);
 void drvshmem_test_singleva_after_fork(void);
 void drvshmem_test_stress(void);
 void drvshmem_test_buddy_basic(void);
+void drvshmem_test_slab_basic(void);
 
 /* test arrays: */
 extern odp_testinfo_t drvshmem_suite[];

[lng-odp] [API-NEXT PATCHv7 6/6] test: drv: shm: adding buddy allocation stress tests

[Christophe Milard]

Stress tests for the random size allocator (buddy allocator in
linux-generic) are added here.

Signed-off-by: Christophe Milard 
---
 .../common_plat/validation/drv/drvshmem/drvshmem.c | 177 +
 .../common_plat/validation/drv/drvshmem/drvshmem.h |   1 +
 2 files changed, 178 insertions(+)

diff --git a/test/common_plat/validation/drv/drvshmem/drvshmem.c 
b/test/common_plat/validation/drv/drvshmem/drvshmem.c
index d4dedea..0f882ae 100644
--- a/test/common_plat/validation/drv/drvshmem/drvshmem.c
+++ b/test/common_plat/validation/drv/drvshmem/drvshmem.c
@@ -938,6 +938,182 @@ void drvshmem_test_slab_basic(void)
odpdrv_shm_pool_destroy(pool);
 }
 
+/*
+ * thread part for the drvshmem_test_buddy_stress
+ */
+static int run_test_buddy_stress(void *arg ODP_UNUSED)
+{
+   odpdrv_shm_t shm;
+   odpdrv_shm_pool_t pool;
+   uint8_t *address;
+   shared_test_data_t *glob_data;
+   uint8_t random_bytes[STRESS_RANDOM_SZ];
+   uint32_t index;
+   uint32_t size;
+   uint8_t data;
+   uint32_t iter;
+   uint32_t i;
+
+   shm = odpdrv_shm_lookup_by_name(MEM_NAME);
+   glob_data = odpdrv_shm_addr(shm);
+   CU_ASSERT_PTR_NOT_NULL(glob_data);
+
+   /* get the pool to test */
+   pool = odpdrv_shm_pool_lookup(POOL_NAME);
+
+   /* wait for general GO! */
+   odpdrv_barrier_wait(&glob_data->test_barrier1);
+   /*
+
+* at each iteration: pick up a random index for
+* glob_data->stress[index]: If the entry is free, allocated small mem
+* randomly. If it is already allocated, make checks and free it:
+* Note that different tread can allocate or free a given block
+*/
+   for (iter = 0; iter < STRESS_ITERATION; iter++) {
+   /* get 4 random bytes from which index, size ,align, flags
+* and data will be derived:
+*/
+   odp_random_data(random_bytes, STRESS_RANDOM_SZ, 0);
+   index = random_bytes[0] & (STRESS_SIZE - 1);
+
+   odp_spinlock_lock(&glob_data->stress_lock);
+
+   switch (glob_data->stress[index].state) {
+   case STRESS_FREE:
+   /* allocated a new block for this entry */
+
+   glob_data->stress[index].state = STRESS_BUSY;
+   odp_spinlock_unlock(&glob_data->stress_lock);
+
+   size  = (random_bytes[1] + 1) << 4; /* up to 4Kb */
+   data  = random_bytes[2];
+
+   address = odpdrv_shm_pool_alloc(pool, size);
+   glob_data->stress[index].address = address;
+   if (address == NULL) { /* out of mem ? */
+   odp_spinlock_lock(&glob_data->stress_lock);
+   glob_data->stress[index].state = STRESS_ALLOC;
+   odp_spinlock_unlock(&glob_data->stress_lock);
+   continue;
+   }
+
+   glob_data->stress[index].size = size;
+   glob_data->stress[index].data_val = data;
+
+   /* write some data: */
+   for (i = 0; i < size; i++)
+   address[i] = (data++) & 0xFF;
+   odp_spinlock_lock(&glob_data->stress_lock);
+   glob_data->stress[index].state = STRESS_ALLOC;
+   odp_spinlock_unlock(&glob_data->stress_lock);
+
+   break;
+
+   case STRESS_ALLOC:
+   /* free the block for this entry */
+
+   glob_data->stress[index].state = STRESS_BUSY;
+   odp_spinlock_unlock(&glob_data->stress_lock);
+   address = glob_data->stress[index].address;
+
+   if (shm == NULL) { /* out of mem ? */
+   odp_spinlock_lock(&glob_data->stress_lock);
+   glob_data->stress[index].state = STRESS_FREE;
+   odp_spinlock_unlock(&glob_data->stress_lock);
+   continue;
+   }
+
+   /* check that data is reachable and correct: */
+   data = glob_data->stress[index].data_val;
+   size = glob_data->stress[index].size;
+   for (i = 0; i < size; i++) {
+   CU_ASSERT(address[i] == (data & 0xFF));
+   data++;
+   }
+
+   odpdrv_shm_pool_free(pool, address);
+
+   odp_spinlock_lock(&glob_data->stress_lock);
+   glob_data->stress[index].state = STRESS_FREE;
+   odp_spinlock_unlock(&glob_data->stress_lock);
+
+   break;
+
+   case STRESS_BUSY:
+   default:
+   

[lng-odp] [API-NEXT PATCHv7 1/6] linux-gen: _ishm: adding buddy and slab allocation

[Christophe Milard]

_ishm now provides functions to create/destroy pools for buddy/slab
memory allocation, as well as functions to allocated/release memory
from the created pools.

Signed-off-by: Christophe Milard 
---
 platform/linux-generic/Makefile.am |   2 +
 platform/linux-generic/_ishm.c |  14 +-
 platform/linux-generic/_ishmpool.c | 811 +
 .../linux-generic/include/_ishmpool_internal.h |  56 ++
 4 files changed, 882 insertions(+), 1 deletion(-)
 create mode 100644 platform/linux-generic/_ishmpool.c
 create mode 100644 platform/linux-generic/include/_ishmpool_internal.h

diff --git a/platform/linux-generic/Makefile.am 
b/platform/linux-generic/Makefile.am
index 6bbe775..d615088 100644
--- a/platform/linux-generic/Makefile.am
+++ b/platform/linux-generic/Makefile.am
@@ -127,6 +127,7 @@ noinst_HEADERS = \
  ${srcdir}/include/_fdserver_internal.h \
  ${srcdir}/include/_ishm_internal.h \
  ${srcdir}/include/_ishmphy_internal.h \
+ ${srcdir}/include/_ishmpool_internal.h \
  ${srcdir}/include/odp_align_internal.h \
  ${srcdir}/include/odp_atomic_internal.h \
  ${srcdir}/include/odp_buffer_inlines.h \
@@ -172,6 +173,7 @@ __LIB__libodp_linux_la_SOURCES = \
   _fdserver.c \
   _ishm.c \
   _ishmphy.c \
+  _ishmpool.c \
   odp_atomic.c \
   odp_barrier.c \
   odp_bitmap.c \
diff --git a/platform/linux-generic/_ishm.c b/platform/linux-generic/_ishm.c
index 23f620d..4c2578b 100644
--- a/platform/linux-generic/_ishm.c
+++ b/platform/linux-generic/_ishm.c
@@ -59,6 +59,7 @@
 #include <_fdserver_internal.h>
 #include <_ishm_internal.h>
 #include <_ishmphy_internal.h>
+#include <_ishmpool_internal.h>
 #include 
 #include 
 #include 
@@ -1441,8 +1442,19 @@ int _odp_ishm_init_global(void)
 * is performed for the main thread... Many init_global() functions
 * indeed assume the availability of odp_shm_reserve()...:
 */
-   return do_odp_ishm_init_local();
+   if (do_odp_ishm_init_local()) {
+   ODP_ERR("unable to init the main thread\n.");
+   goto init_glob_err4;
+   }
+
+   /* get ready to create pools: */
+   _odp_ishm_pool_init();
 
+   return 0;
+
+init_glob_err4:
+   if (_odp_ishmphy_unbook_va())
+   ODP_ERR("unable to unbook virtual space\n.");
 init_glob_err3:
if (munmap(ishm_ftbl, sizeof(ishm_ftable_t)) < 0)
ODP_ERR("unable to munmap main fragment table\n.");
diff --git a/platform/linux-generic/_ishmpool.c 
b/platform/linux-generic/_ishmpool.c
new file mode 100644
index 000..df6e49e
--- /dev/null
+++ b/platform/linux-generic/_ishmpool.c
@@ -0,0 +1,811 @@
+/* Copyright (c) 2017, Linaro Limited
+ * All rights reserved.
+ *
+ * SPDX-License-Identifier: BSD-3-Clause
+ */
+
+/* This file gathers the buddy and slab allocation functionality provided
+ * by _ishm.
+ * _odp_ishmpool_create() can be used to create a pool for buddy/slab
+ * allocation. _odp_ishmpool_create() will allocate a memory area using
+ * ishm_reserve() for both the control part (needed for tracking
+ * allocation/free...) and the user memory itself (part of which will be given
+ * at each ishmpool_alloc()).
+ * The element size provided at pool creation time determines whether
+ * to pool will of type buddy or slab.
+ * For buddy, all allocations are rounded to the nearest power of 2.
+ *
+ * The implementation of the buddy allocator is very traditional: it
+ * maintains N lists of free buffers.
+ * The control part actually contains these N queue heads, (N-M are actually
+ * used), the free buffers themselves being used for chaining (the chaining 
info
+ * is in the buffers: as they are "free" they should not be touched by the
+ * user). The control part also contains a array of bytes for remembering
+ * the size (actually the order) of the allocated buffers:
+ * There are 2^(N-M) such bytes, this number being the maximum number of
+ * allocated buffers (when all allocation are <= 2^M bytes)
+ * Buddy allocators handle fragmentation by splitting or merging blocks by 2.
+ * They guarantee a minimum efficiency of 50%, at worse case fragmentation.
+ *
+ * Slab implementation is even simpler, all free elements being queued in
+ * one single queue at init, taken from this queue when allocated and
+ * returned to this same queue when freed.
+ *
+ * The reason for not using malloc() is that malloc does not guarantee
+ * memory sharability between ODP threads (regardless of their implememtation)
+ * which ishm_reserve() can do. see the comments around
+ * _odp_ishmbud_pool_create() and ishm_reserve() for more details.
+ *
+ * This file is divided in 3 sections: the first one regroups functions
+ * needed by 

Re: [lng-odp] [API-NEXT PATCHv7 1/6] linux-gen: _ishm: adding buddy and slab allocation

[Yi He]

For this patch series:

Reviewed-and-tested-by: Yi He 



On 13 January 2017 at 15:55, Christophe Milard  wrote:

> _ishm now provides functions to create/destroy pools for buddy/slab
> memory allocation, as well as functions to allocated/release memory
> from the created pools.
>
> Signed-off-by: Christophe Milard 
> ---
>  platform/linux-generic/Makefile.am |   2 +
>  platform/linux-generic/_ishm.c |  14 +-
>  platform/linux-generic/_ishmpool.c | 811
> +
>  .../linux-generic/include/_ishmpool_internal.h |  56 ++
>  4 files changed, 882 insertions(+), 1 deletion(-)
>  create mode 100644 platform/linux-generic/_ishmpool.c
>  create mode 100644 platform/linux-generic/include/_ishmpool_internal.h
>
> diff --git a/platform/linux-generic/Makefile.am b/platform/linux-generic/
> Makefile.am
> index 6bbe775..d615088 100644
> --- a/platform/linux-generic/Makefile.am
> +++ b/platform/linux-generic/Makefile.am
> @@ -127,6 +127,7 @@ noinst_HEADERS = \
>   ${srcdir}/include/_fdserver_internal.h \
>   ${srcdir}/include/_ishm_internal.h \
>   ${srcdir}/include/_ishmphy_internal.h \
> + ${srcdir}/include/_ishmpool_internal.h \
>   ${srcdir}/include/odp_align_internal.h \
>   ${srcdir}/include/odp_atomic_internal.h \
>   ${srcdir}/include/odp_buffer_inlines.h \
> @@ -172,6 +173,7 @@ __LIB__libodp_linux_la_SOURCES = \
>_fdserver.c \
>_ishm.c \
>_ishmphy.c \
> +  _ishmpool.c \
>odp_atomic.c \
>odp_barrier.c \
>odp_bitmap.c \
> diff --git a/platform/linux-generic/_ishm.c b/platform/linux-generic/_
> ishm.c
> index 23f620d..4c2578b 100644
> --- a/platform/linux-generic/_ishm.c
> +++ b/platform/linux-generic/_ishm.c
> @@ -59,6 +59,7 @@
>  #include <_fdserver_internal.h>
>  #include <_ishm_internal.h>
>  #include <_ishmphy_internal.h>
> +#include <_ishmpool_internal.h>
>  #include 
>  #include 
>  #include 
> @@ -1441,8 +1442,19 @@ int _odp_ishm_init_global(void)
>  * is performed for the main thread... Many init_global() functions
>  * indeed assume the availability of odp_shm_reserve()...:
>  */
> -   return do_odp_ishm_init_local();
> +   if (do_odp_ishm_init_local()) {
> +   ODP_ERR("unable to init the main thread\n.");
> +   goto init_glob_err4;
> +   }
> +
> +   /* get ready to create pools: */
> +   _odp_ishm_pool_init();
>
> +   return 0;
> +
> +init_glob_err4:
> +   if (_odp_ishmphy_unbook_va())
> +   ODP_ERR("unable to unbook virtual space\n.");
>  init_glob_err3:
> if (munmap(ishm_ftbl, sizeof(ishm_ftable_t)) < 0)
> ODP_ERR("unable to munmap main fragment table\n.");
> diff --git a/platform/linux-generic/_ishmpool.c b/platform/linux-generic/_
> ishmpool.c
> new file mode 100644
> index 000..df6e49e
> --- /dev/null
> +++ b/platform/linux-generic/_ishmpool.c
> @@ -0,0 +1,811 @@
> +/* Copyright (c) 2017, Linaro Limited
> + * All rights reserved.
> + *
> + * SPDX-License-Identifier: BSD-3-Clause
> + */
> +
> +/* This file gathers the buddy and slab allocation functionality provided
> + * by _ishm.
> + * _odp_ishmpool_create() can be used to create a pool for buddy/slab
> + * allocation. _odp_ishmpool_create() will allocate a memory area using
> + * ishm_reserve() for both the control part (needed for tracking
> + * allocation/free...) and the user memory itself (part of which will be
> given
> + * at each ishmpool_alloc()).
> + * The element size provided at pool creation time determines whether
> + * to pool will of type buddy or slab.
> + * For buddy, all allocations are rounded to the nearest power of 2.
> + *
> + * The implementation of the buddy allocator is very traditional: it
> + * maintains N lists of free buffers.
> + * The control part actually contains these N queue heads, (N-M are
> actually
> + * used), the free buffers themselves being used for chaining (the
> chaining info
> + * is in the buffers: as they are "free" they should not be touched by the
> + * user). The control part also contains a array of bytes for remembering
> + * the size (actually the order) of the allocated buffers:
> + * There are 2^(N-M) such bytes, this number being the maximum number of
> + * allocated buffers (when all allocation are <= 2^M bytes)
> + * Buddy allocators handle fragmentation by splitting or merging blocks
> by 2.
> + * They guarantee a minimum efficiency of 50%, at worse case
> fragmentation.
> + *
> + * Slab implementation is even simpler, all free elements being queued in
> + * one single queue at init, taken from this queue when allocated and
> + * returned to this same queue when freed.
> + *
> + * The reason for not usi

Re: [lng-odp] [API-NEXT PATCHv6 1/6] linux-gen: _ishm: adding buddy and slab allocation

[Yi He]

Thanks Christophe for the explanation

Every ODP thread can individually creates ishmpools for small memory
allocation, that's convenient.
I'll read into platform/linux-generic/_ishm.c. as you suggested to
understand the impl.

Thanks and best regards, Yi

On 12 January 2017 at 18:50, Christophe Milard  wrote:

> _ishm now provides functions to create/destroy pools for buddy/slab
> memory allocation, as well as functions to allocated/release memory
> from the created pools.
>
> Signed-off-by: Christophe Milard 
> ---
>  platform/linux-generic/Makefile.am |   2 +
>  platform/linux-generic/_ishm.c |  14 +-
>  platform/linux-generic/_ishmpool.c | 811
> +
>  .../linux-generic/include/_ishmpool_internal.h |  56 ++
>  4 files changed, 882 insertions(+), 1 deletion(-)
>  create mode 100644 platform/linux-generic/_ishmpool.c
>  create mode 100644 platform/linux-generic/include/_ishmpool_internal.h
>
> diff --git a/platform/linux-generic/Makefile.am b/platform/linux-generic/
> Makefile.am
> index 999a7f5..d153c5d 100644
> --- a/platform/linux-generic/Makefile.am
> +++ b/platform/linux-generic/Makefile.am
> @@ -127,6 +127,7 @@ noinst_HEADERS = \
>   ${srcdir}/include/_fdserver_internal.h \
>   ${srcdir}/include/_ishm_internal.h \
>   ${srcdir}/include/_ishmphy_internal.h \
> + ${srcdir}/include/_ishmpool_internal.h \
>   ${srcdir}/include/odp_align_internal.h \
>   ${srcdir}/include/odp_atomic_internal.h \
>   ${srcdir}/include/odp_buffer_inlines.h \
> @@ -171,6 +172,7 @@ __LIB__libodp_linux_la_SOURCES = \
>_fdserver.c \
>_ishm.c \
>_ishmphy.c \
> +  _ishmpool.c \
>odp_atomic.c \
>odp_barrier.c \
>odp_buffer.c \
> diff --git a/platform/linux-generic/_ishm.c b/platform/linux-generic/_
> ishm.c
> index 23f620d..4c2578b 100644
> --- a/platform/linux-generic/_ishm.c
> +++ b/platform/linux-generic/_ishm.c
> @@ -59,6 +59,7 @@
>  #include <_fdserver_internal.h>
>  #include <_ishm_internal.h>
>  #include <_ishmphy_internal.h>
> +#include <_ishmpool_internal.h>
>  #include 
>  #include 
>  #include 
> @@ -1441,8 +1442,19 @@ int _odp_ishm_init_global(void)
>  * is performed for the main thread... Many init_global() functions
>  * indeed assume the availability of odp_shm_reserve()...:
>  */
> -   return do_odp_ishm_init_local();
> +   if (do_odp_ishm_init_local()) {
> +   ODP_ERR("unable to init the main thread\n.");
> +   goto init_glob_err4;
> +   }
> +
> +   /* get ready to create pools: */
> +   _odp_ishm_pool_init();
>
> +   return 0;
> +
> +init_glob_err4:
> +   if (_odp_ishmphy_unbook_va())
> +   ODP_ERR("unable to unbook virtual space\n.");
>  init_glob_err3:
> if (munmap(ishm_ftbl, sizeof(ishm_ftable_t)) < 0)
> ODP_ERR("unable to munmap main fragment table\n.");
> diff --git a/platform/linux-generic/_ishmpool.c b/platform/linux-generic/_
> ishmpool.c
> new file mode 100644
> index 000..df6e49e
> --- /dev/null
> +++ b/platform/linux-generic/_ishmpool.c
> @@ -0,0 +1,811 @@
> +/* Copyright (c) 2017, Linaro Limited
> + * All rights reserved.
> + *
> + * SPDX-License-Identifier: BSD-3-Clause
> + */
> +
> +/* This file gathers the buddy and slab allocation functionality provided
> + * by _ishm.
> + * _odp_ishmpool_create() can be used to create a pool for buddy/slab
> + * allocation. _odp_ishmpool_create() will allocate a memory area using
> + * ishm_reserve() for both the control part (needed for tracking
> + * allocation/free...) and the user memory itself (part of which will be
> given
> + * at each ishmpool_alloc()).
> + * The element size provided at pool creation time determines whether
> + * to pool will of type buddy or slab.
> + * For buddy, all allocations are rounded to the nearest power of 2.
> + *
> + * The implementation of the buddy allocator is very traditional: it
> + * maintains N lists of free buffers.
> + * The control part actually contains these N queue heads, (N-M are
> actually
> + * used), the free buffers themselves being used for chaining (the
> chaining info
> + * is in the buffers: as they are "free" they should not be touched by the
> + * user). The control part also contains a array of bytes for remembering
> + * the size (actually the order) of the allocated buffers:
> + * There are 2^(N-M) such bytes, this number being the maximum number of
> + * allocated buffers (when all allocation are <= 2^M bytes)
> + * Buddy allocators handle fragmentation by splitting or merging blocks
> by 2.
> + * They guarantee a minimum efficiency of 50%, at worse case
> fragmentation.
> + *
> + * Slab implementation is ev

Re: [lng-odp] [API-NEXT PATCHv6 1/6] linux-gen: _ishm: adding buddy and slab allocation

[Christophe Milard]

Just note that the pool stuff is only exposed to the south (DRV)
interface so far, as I need it there. I actually raised the case (a
long time ago) for having it on the north side with very little
traction.
As I mentioned, I think this will change if we start caring about the
"process-mode" as I expect many current "mallocs" to be expecteded to
be visible amoung ODP thread... But today we assume pthreads
everywhere.

It is trivial to add this to the north side

Christophe

On 13 January 2017 at 08:36, Yi He  wrote:
> Thanks Christophe for the explanation
>
> Every ODP thread can individually creates ishmpools for small memory
> allocation, that's convenient.
> I'll read into platform/linux-generic/_ishm.c. as you suggested to
> understand the impl.
>
> Thanks and best regards, Yi
>
> On 12 January 2017 at 18:50, Christophe Milard
>  wrote:
>>
>> _ishm now provides functions to create/destroy pools for buddy/slab
>> memory allocation, as well as functions to allocated/release memory
>> from the created pools.
>>
>> Signed-off-by: Christophe Milard 
>> ---
>>  platform/linux-generic/Makefile.am |   2 +
>>  platform/linux-generic/_ishm.c |  14 +-
>>  platform/linux-generic/_ishmpool.c | 811
>> +
>>  .../linux-generic/include/_ishmpool_internal.h |  56 ++
>>  4 files changed, 882 insertions(+), 1 deletion(-)
>>  create mode 100644 platform/linux-generic/_ishmpool.c
>>  create mode 100644 platform/linux-generic/include/_ishmpool_internal.h
>>
>> diff --git a/platform/linux-generic/Makefile.am
>> b/platform/linux-generic/Makefile.am
>> index 999a7f5..d153c5d 100644
>> --- a/platform/linux-generic/Makefile.am
>> +++ b/platform/linux-generic/Makefile.am
>> @@ -127,6 +127,7 @@ noinst_HEADERS = \
>>   ${srcdir}/include/_fdserver_internal.h \
>>   ${srcdir}/include/_ishm_internal.h \
>>   ${srcdir}/include/_ishmphy_internal.h \
>> + ${srcdir}/include/_ishmpool_internal.h \
>>   ${srcdir}/include/odp_align_internal.h \
>>   ${srcdir}/include/odp_atomic_internal.h \
>>   ${srcdir}/include/odp_buffer_inlines.h \
>> @@ -171,6 +172,7 @@ __LIB__libodp_linux_la_SOURCES = \
>>_fdserver.c \
>>_ishm.c \
>>_ishmphy.c \
>> +  _ishmpool.c \
>>odp_atomic.c \
>>odp_barrier.c \
>>odp_buffer.c \
>> diff --git a/platform/linux-generic/_ishm.c
>> b/platform/linux-generic/_ishm.c
>> index 23f620d..4c2578b 100644
>> --- a/platform/linux-generic/_ishm.c
>> +++ b/platform/linux-generic/_ishm.c
>> @@ -59,6 +59,7 @@
>>  #include <_fdserver_internal.h>
>>  #include <_ishm_internal.h>
>>  #include <_ishmphy_internal.h>
>> +#include <_ishmpool_internal.h>
>>  #include 
>>  #include 
>>  #include 
>> @@ -1441,8 +1442,19 @@ int _odp_ishm_init_global(void)
>>  * is performed for the main thread... Many init_global()
>> functions
>>  * indeed assume the availability of odp_shm_reserve()...:
>>  */
>> -   return do_odp_ishm_init_local();
>> +   if (do_odp_ishm_init_local()) {
>> +   ODP_ERR("unable to init the main thread\n.");
>> +   goto init_glob_err4;
>> +   }
>> +
>> +   /* get ready to create pools: */
>> +   _odp_ishm_pool_init();
>>
>> +   return 0;
>> +
>> +init_glob_err4:
>> +   if (_odp_ishmphy_unbook_va())
>> +   ODP_ERR("unable to unbook virtual space\n.");
>>  init_glob_err3:
>> if (munmap(ishm_ftbl, sizeof(ishm_ftable_t)) < 0)
>> ODP_ERR("unable to munmap main fragment table\n.");
>> diff --git a/platform/linux-generic/_ishmpool.c
>> b/platform/linux-generic/_ishmpool.c
>> new file mode 100644
>> index 000..df6e49e
>> --- /dev/null
>> +++ b/platform/linux-generic/_ishmpool.c
>> @@ -0,0 +1,811 @@
>> +/* Copyright (c) 2017, Linaro Limited
>> + * All rights reserved.
>> + *
>> + * SPDX-License-Identifier: BSD-3-Clause
>> + */
>> +
>> +/* This file gathers the buddy and slab allocation functionality provided
>> + * by _ishm.
>> + * _odp_ishmpool_create() can be used to create a pool for buddy/slab
>> + * allocation. _odp_ishmpool_create() will allocate a memory area using
>> + * ishm_reserve() for both the control part (needed for tracking
>> + * allocation/free...) and the user memory itself (part of which will be
>> given
>> + * at each ishmpool_alloc()).
>> + * The element size provided at pool creation time determines whether
>> + * to pool will of type buddy or slab.
>> + * For buddy, all allocations are rounded to the nearest power of 2.
>> + *
>> + * The implementation of the buddy allocator is very traditional: it
>> + * maintains N lists of free buffers.
>> + * The control part actually contains these N queue heads, (N-M are
>> actually
>>