[lng-odp] [PATCHv7 3/3] configure.ac update version numbers

[Maxim Uvarov]

Default is abi compat mode, all interface functions changed,
so increase first number of .so

Signed-off-by: Maxim Uvarov 
---
 configure.ac | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/configure.ac b/configure.ac
index b460a65..e7b8f8a 100644
--- a/configure.ac
+++ b/configure.ac
@@ -3,7 +3,7 @@ AC_PREREQ([2.5])
 # Set correct API version
 ##
 m4_define([odpapi_generation_version], [1])
-m4_define([odpapi_major_version], [11])
+m4_define([odpapi_major_version], [12])
 m4_define([odpapi_minor_version], [0])
 m4_define([odpapi_point_version], [0])
 m4_define([odpapi_version],
@@ -30,10 +30,10 @@ AM_SILENT_RULES([yes])
 ##
 # Set correct platform library version
 ##
-ODP_LIBSO_VERSION=111:0:0
+ODP_LIBSO_VERSION=112:0:0
 AC_SUBST(ODP_LIBSO_VERSION)
 
-ODPHELPER_LIBSO_VERSION=110:0:1
+ODPHELPER_LIBSO_VERSION=112:0:0
 AC_SUBST(ODPHELPER_LIBSO_VERSION)
 
 # Checks for programs.
-- 
2.7.1.250.gff4ea60

[lng-odp] [PATCHv7 2/3] changelog: summary of changes for odp v1.12.0.0

[Maxim Uvarov]

From: Bill Fischofer 

Signed-off-by: Bill Fischofer 
Signed-off-by: Maxim Uvarov 
---
 CHANGELOG | 177 ++
 1 file changed, 177 insertions(+)

diff --git a/CHANGELOG b/CHANGELOG
index 1d652a8..b125d63 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -1,3 +1,180 @@
+== OpenDataPlane (1.12.0.0)
+
+=== New Features
+
+ APIs
+ODP v1.12.0.0 has no API changes from previous v1.11.0 Monarch LTS. Version
+is increased in current development release to make room for Monarch updates
+numbers.
+
+ Application Binary Interface (ABI) Support
+Support is added to enable ODP applications to be binary compatible across
+different implementations of ODP sharing the same Instruction Set Architecture
+(ISA). This support introduces a new `configure` option:
+
+`--enable-abi-compat=yes`::
+This is the default and specifies that the ODP library is to be built to
+support ABI compatibility mode. In this mode ODP APIs are never inlined. ABI
+compatibility ensures maximum application portability in cloud environments.
+
+`--enable-abi-compat=no`::
+Specify this option to enable the inlining of ODP APIs. This may result in
+improved performance at the cost of ABI compatibility and is suitable for
+applications running in embedded environments.
+
+Note that ODP applications retain source code portability between ODP
+implementations regardless of the ABI mode chosen. To move to a different ODP
+application running on a different ISA, code need simply be recompiled against
+that target ODP implementation.
+
+ SCTP Parsing Support
+The ODP classifier adds support for recognizing Stream Control Transmission
+Protocol (SCTP) packets. The APIs for this were previously not implemented.
+
+=== Packaging and Implementation Refinements
+
+ Remove dependency on Linux headers
+ODP no longer has a dependency on Linux headers. This will help make the
+odp-linux reference implementation more easily portable to non-Linux
+environments.
+
+ Remove dependency on helpers
+The odp-linux implementation has been made independent of the helper library
+to avoid circular dependency issues with packaging. Helper functions may use
+ODP APIs, however ODP implementations should not use helper functions.
+
+ Reorganization of `test` directory
+The `test` directory has been reorganized to better support a unified approach
+to ODP component testing. API tests now live in
+`test/common_plat/validation/api` instead of the former
+`test/validation`. With this change performance and validation tests, as well
+as common and platform-specific tests can all be part of a unified test
+hierarchy.
+
+The resulting test tree now looks like:
+
+.New `test` directory hierarchy
+-
+test
+├── common_plat
+│   ├── common
+│   ├── m4
+│   ├── miscellaneous
+│   ├── performance
+│   └── validation
+│   └── api
+│   ├── atomic
+│   ├── barrier
+│   ├── buffer
+│   ├── classification
+│   ├── cpumask
+│   ├── crypto
+│   ├── errno
+│   ├── hash
+│   ├── init
+│   ├── lock
+│   ├── packet
+│   ├── pktio
+│   ├── pool
+│   ├── queue
+│   ├── random
+│   ├── scheduler
+│   ├── shmem
+│   ├── std_clib
+│   ├── system
+│   ├── thread
+│   ├── time
+│   ├── timer
+│   └── traffic_mngr
+├── linux-generic
+│   ├── m4
+│   ├── mmap_vlan_ins
+│   ├── performance
+│   ├── pktio_ipc
+│   ├── ring
+│   └── validation
+│   └── api
+│   ├── pktio
+│   └── shmem
+└── m4
+-
+
+ Pools
+The maximum number of pools that may be created in the odp-linux reference
+implementation has been raised from 16 to 64.
+
+ Upgrade to DPDK 16.07
+The DPDK pktio support in odp-linux has been upgraded to work with DPDK 16.07.
+A number of miscellaneous fixes and performance improvements in this support
+are also present.
+
+ PktIO TAP Interface Classifier Support
+Packet I/O interfaces operating in TAP mode now can feed packets to the ODP
+classifier the same as other pktio modes can do.
+
+=== Performance Improvements
+
+ Burst-mode buffer allocation
+The scheduler and pktio components have been reworked to use burst-mode
+buffer allocation/deallocation, yielding a measurable performance gain in
+almost all cases.
+
+ Burst-mode queue operations
+ODP queues internally now attempt to use burst-mode enq/deq operations to
+accelerate performance where applicable.
+
+ Ring-based Scheduler Priority Queues
+The ODP scheduler has been enhanced to use ring-based priority queues, 
resulting
+in significantly better scalability in many core environments.
+
+ GitHub Automation Support
+ODP now supports the Travis framework needed to trigger CI automation in
+conjunction with GitHub. This support is considered experimental for now.

[lng-odp] [PATCHv7 1/3] configure.ac: disable shared library for non abi compat mode

[Maxim Uvarov]

original configure.ac enables abi compat mode by default,
even without --enable-abi-compat provided. And has broken
logic to disable abi compat mode. Correct logic to build abi
compat mode and option to disable it. Shared library is not
needed for non abi compat mode, so turn it off.

Signed-off-by: Maxim Uvarov 
---
 configure.ac | 9 -
 1 file changed, 4 insertions(+), 5 deletions(-)

diff --git a/configure.ac b/configure.ac
index be5a292..b460a65 100644
--- a/configure.ac
+++ b/configure.ac
@@ -241,13 +241,11 @@ ODP_CFLAGS="$ODP_CFLAGS -DODP_DEBUG=$ODP_DEBUG"
 ODP_ABI_COMPAT=1
 abi_compat=yes
 AC_ARG_ENABLE([abi-compat],
-[  --enable-abi-compat build all targets in ABI compatible mode 
(default=yes)],
-[if test "x$enableval" = "xyes"; then
-   ODP_ABI_COMPAT=1
-   abi_compat=yes
- else
+[  --disable-abi-compat disables ABI compatible mode, enables inline 
code in header files],
+[if test "x$enableval" = "xno"; then
ODP_ABI_COMPAT=0
abi_compat=no
+   enable_shared=no
 fi])
 AC_SUBST(ODP_ABI_COMPAT)
 
@@ -336,6 +334,7 @@ AC_MSG_RESULT([
static libraries:   ${enable_static}
shared libraries:   ${enable_shared}
ABI compatible: ${abi_compat}
+   ODP_ABI_COMPAT: ${ODP_ABI_COMPAT}
cunit:  ${cunit_support}
test_vald:  ${test_vald}
test_perf:  ${test_perf}
-- 
2.7.1.250.gff4ea60

[lng-odp] [PATCHv7 0/3] changelog: summary of changes for odp v1.12.0.0

[Maxim Uvarov]

v7: more accurate define disable option (Anders)

Bill Fischofer (1):
  changelog: summary of changes for odp v1.12.0.0

Maxim Uvarov (2):
  configure.ac: disable shared library for non abi compat mode
  configure.ac update version numbers

 CHANGELOG| 177 +++
 configure.ac |  15 +++--
 2 files changed, 184 insertions(+), 8 deletions(-)

-- 
2.7.1.250.gff4ea60

Re: [lng-odp] [PATCHv6 1/3] configure.ac: disable shared library for non abi compat mode

[Maxim Uvarov]

On 11/24/16 23:12, Anders Roxell wrote:

On 24 November 2016 at 17:03, Maxim Uvarov  wrote:

original configure.ac enables abi compat mode by default,
even without --enable-abi-compat provided. And has broken
logic to disable abi compat mode. Correct logic to build abi
compat mode and option to disable it. Shared library is not
needed for non abi compat mode, so turn it off.

Signed-off-by: Maxim Uvarov 
---
  configure.ac | 9 ++---
  1 file changed, 6 insertions(+), 3 deletions(-)

diff --git a/configure.ac b/configure.ac
index be5a292..7cb0c7a 100644
--- a/configure.ac
+++ b/configure.ac
@@ -241,16 +241,19 @@ ODP_CFLAGS="$ODP_CFLAGS -DODP_DEBUG=$ODP_DEBUG"
  ODP_ABI_COMPAT=1
  abi_compat=yes
  AC_ARG_ENABLE([abi-compat],
-[  --enable-abi-compat build all targets in ABI compatible mode 
(default=yes)],
+[  --disable-abi-compatdisable build all targets in ABI compatible 
mode (default=no)],
  [if test "x$enableval" = "xyes"; then
 ODP_ABI_COMPAT=1
-   abi_compat=yes
   else
 ODP_ABI_COMPAT=0
-   abi_compat=no
  fi])
  AC_SUBST(ODP_ABI_COMPAT)

+if test $ODP_ABI_COMPAT -eq 0; then
+   enable_shared=no
+   abi_compat=no
+fi
+
  ##
  # Default warning setup
  ##
--
2.7.1.250.gff4ea60


I think it should look something like this:
$ git df
diff --git a/configure.ac b/configure.ac
index be5a292..7becc2a 100644
--- a/configure.ac
+++ b/configure.ac
@@ -241,13 +241,11 @@ ODP_CFLAGS="$ODP_CFLAGS -DODP_DEBUG=$ODP_DEBUG"
  ODP_ABI_COMPAT=1
  abi_compat=yes
  AC_ARG_ENABLE([abi-compat],
-[  --enable-abi-compat build all targets in ABI compatible
mode (default=yes)],
-[if test "x$enableval" = "xyes"; then
-   ODP_ABI_COMPAT=1
-   abi_compat=yes
- else
+[  --disable-abi-compat disables ABI compatible mode, enables
inline code in header files],
+[if test "x$enableval" = "xno"; then
 ODP_ABI_COMPAT=0
 abi_compat=no
+   enable_shared=no
  fi])
  AC_SUBST(ODP_ABI_COMPAT)

@@ -336,6 +334,7 @@ AC_MSG_RESULT([
 static libraries:   ${enable_static}
 shared libraries:   ${enable_shared}
 ABI compatible: ${abi_compat}
+   ODP_ABI_COMPAT: ${ODP_ABI_COMPAT}
 cunit:  ${cunit_support}
 test_vald:  ${test_vald}
 test_perf:  ${test_perf}

ok -> v7

Re: [lng-odp] [PATCHv6 1/3] configure.ac: disable shared library for non abi compat mode

[Anders Roxell]

On 24 November 2016 at 17:03, Maxim Uvarov  wrote:
> original configure.ac enables abi compat mode by default,
> even without --enable-abi-compat provided. And has broken
> logic to disable abi compat mode. Correct logic to build abi
> compat mode and option to disable it. Shared library is not
> needed for non abi compat mode, so turn it off.
>
> Signed-off-by: Maxim Uvarov 
> ---
>  configure.ac | 9 ++---
>  1 file changed, 6 insertions(+), 3 deletions(-)
>
> diff --git a/configure.ac b/configure.ac
> index be5a292..7cb0c7a 100644
> --- a/configure.ac
> +++ b/configure.ac
> @@ -241,16 +241,19 @@ ODP_CFLAGS="$ODP_CFLAGS -DODP_DEBUG=$ODP_DEBUG"
>  ODP_ABI_COMPAT=1
>  abi_compat=yes
>  AC_ARG_ENABLE([abi-compat],
> -[  --enable-abi-compat build all targets in ABI compatible mode 
> (default=yes)],
> +[  --disable-abi-compatdisable build all targets in ABI compatible 
> mode (default=no)],
>  [if test "x$enableval" = "xyes"; then
> ODP_ABI_COMPAT=1
> -   abi_compat=yes
>   else
> ODP_ABI_COMPAT=0
> -   abi_compat=no
>  fi])
>  AC_SUBST(ODP_ABI_COMPAT)
>
> +if test $ODP_ABI_COMPAT -eq 0; then
> +   enable_shared=no
> +   abi_compat=no
> +fi
> +
>  ##
>  # Default warning setup
>  ##
> --
> 2.7.1.250.gff4ea60
>

I think it should look something like this:
$ git df
diff --git a/configure.ac b/configure.ac
index be5a292..7becc2a 100644
--- a/configure.ac
+++ b/configure.ac
@@ -241,13 +241,11 @@ ODP_CFLAGS="$ODP_CFLAGS -DODP_DEBUG=$ODP_DEBUG"
 ODP_ABI_COMPAT=1
 abi_compat=yes
 AC_ARG_ENABLE([abi-compat],
-[  --enable-abi-compat build all targets in ABI compatible
mode (default=yes)],
-[if test "x$enableval" = "xyes"; then
-   ODP_ABI_COMPAT=1
-   abi_compat=yes
- else
+[  --disable-abi-compat disables ABI compatible mode, enables
inline code in header files],
+[if test "x$enableval" = "xno"; then
ODP_ABI_COMPAT=0
abi_compat=no
+   enable_shared=no
 fi])
 AC_SUBST(ODP_ABI_COMPAT)

@@ -336,6 +334,7 @@ AC_MSG_RESULT([
static libraries:   ${enable_static}
shared libraries:   ${enable_shared}
ABI compatible: ${abi_compat}
+   ODP_ABI_COMPAT: ${ODP_ABI_COMPAT}
cunit:  ${cunit_support}
test_vald:  ${test_vald}
test_perf:  ${test_perf}

Re: [lng-odp] [API-NEXT PATCHv9 00/15] using _ishm as north API mem allocator

[Maxim Uvarov]

Christophe,

with this patch series and without it I see odp_pktio_perf hangs in 
background.
And it looks like accept() never pass. Interesting is that Cunit 
reported this test as

passed and actually it's in background. Other apps looks like ok.

81../sysdeps/unix/syscall-template.S: No such file or directory.
(gdb) bt
#0  0x2b030bcd46c0 in __accept_nocancel () at 
../sysdeps/unix/syscall-template.S:81
#1  0x2b030ba1bfe5 in wait_requests (sock=) at 
_fdserver.c:551

#2  _odp_fdserver_init_global () at _fdserver.c:633
#3  0x2b030ba23c82 in odp_init_global 
(instance=instance@entry=0x7ffdf2199138, params=params@entry=0x0,

platform_params=platform_params@entry=0x0) at odp_init.c:114
#4  0x00401ead in main (argc=1, argv=0x7ffdf2199298) at 
odp_pktio_perf.c:1030

(gdb) up
#1  0x2b030ba1bfe5 in wait_requests (sock=) at 
_fdserver.c:551
551c_socket = accept(sock, (struct sockaddr *), 
_sz);

(gdb) c
Continuing.
^C^C^C^C


$sudo strace -p 31410 -ff
Process 31410 attached
accept(3,

Can you reproduce that? it's not root user.

Maxim.


On 11/24/16 19:22, Christophe Milard wrote:

since v8:
 - commit msg typo fix (Maxim)
 - use snprintf() instead of sprintf() (Maxim)
 - better handling of error cases (Maxim)

since v7:
 - Fix not to use single-va memory when the requirement alignment exactely
matches the page size (Maxim)
 - patch to clean-up all odp--* files in /tmp and in the huge
page mount point at startup (in case some previous killed / crashed
ODP instance did not clean properly.) (Maxim)

since v6:
 - All points according to Petri's request i.e.:
Odp_shm_find_external() changed again: now odp_shm_import().
Function description updated.

since v5:
 -fixed doxygen issue (Bill)

since v4:
 - All points of v3 according to Petri's request i.e.:
-All API patches merges as one single patch.
-lock flag removed: memory is now always locked.
-alignement and flag inherited from remote ODP instance when sharing
 memory.
-function name changed to: odp_shm_find_external().

since v3:
-Comments from Petri addressed, with exceptions for:
 -Patches related to the API not merged as they are part of different
  functions.
 -Lock flag not removed: Why wouldn't any ODP application being allowed
  to reserve memory for slower (control or other things) businnes.
  Should really ALL shared memory bee locked?
 -Alignment and flag parameter of odp_shm_reserve_external() kept:
  The alignment (and some flags) affect virtual memory, which is different
  between for each ODP instance.
 -function name odp_shm_reserve_external() kept, as this function does
  return a new handle (which requires free() as any handle would)
These 4 items are sheduled for arch call today

since v2:
-some minor changes on the doc (Bill)
-doc for the ODP_SHM_EXPORT flag (Christophe)
-reduction of memory reguirement for shm tests (Mattias)

since v1:
-flag _ODP_SHM_PROC_NOCREAT and _ODP_SHM_O_EXCL get new values
(but remain useless: Should be removed when IPC is updated)  (Maxim)

-In get_ishm_flags(), odp_shm_capability() local variable flgs renamed
(for be better distinction from other "flags" variable. (Maxim)

-Added doc updates with shm api extensions.

This Patch series aims at using _ishm as north API memory allocator.
odp_shared_memory.c just becomes a wrapper around _ishm.
_ishm supports "process mode", i.e. memory allocated  with _ishm
is sharable by all ODP threads of a given ODP instance, regardless of
thread type (e.g. process) or thread creation time (for time).

NOTE: This patch series will break IPC: This is due to the fact that
IPC relied on a "special case" in the former memory allocator that broke
ODP instance scoping. I don't think this should be kept.
I have included in this patch series a function to share memory between
designated ODP instances. If we want to have IPC, it should use that.

Christophe Milard (15):
   linux-gen: _ishm: fix for alignment request matching page size
   linux-gen: init: removing possible obsolete ODP files at startup
   linux-gen: _ishm: create description file for external memory sharing
   linux-gen: _ishm: allow memory alloc/free at global init/term
   linux-gen: use ishm as north API mem allocator
   linux-gen: push internal flag definition
   api: shm: add flags to shm_reserve and function to find external mem
   linux-gen: shm: new ODP_SHM_SINGLE_VA flag implementation
   test: api: shmem: new proper tests for shm API
   linux-gen: _ishm: adding function to map memory from other ODP
   linux-gen: shm: add flag and function to share memory between ODP
 instances
   test: linux-gen: api: shmem: test sharing memory between ODP instances
   linux-gen: _ishm: cleaning remaining block at odp_term_global
   linux_gen: _ishm: decreasing the number of error messages when no huge
 pages
   doc: updating 

[lng-odp] [PATCHv6 2/3] changelog: summary of changes for odp v1.12.0.0

[Maxim Uvarov]

From: Bill Fischofer 

Signed-off-by: Bill Fischofer 
Signed-off-by: Maxim Uvarov 
---
 CHANGELOG | 177 ++
 1 file changed, 177 insertions(+)

diff --git a/CHANGELOG b/CHANGELOG
index 1d652a8..b125d63 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -1,3 +1,180 @@
+== OpenDataPlane (1.12.0.0)
+
+=== New Features
+
+ APIs
+ODP v1.12.0.0 has no API changes from previous v1.11.0 Monarch LTS. Version
+is increased in current development release to make room for Monarch updates
+numbers.
+
+ Application Binary Interface (ABI) Support
+Support is added to enable ODP applications to be binary compatible across
+different implementations of ODP sharing the same Instruction Set Architecture
+(ISA). This support introduces a new `configure` option:
+
+`--enable-abi-compat=yes`::
+This is the default and specifies that the ODP library is to be built to
+support ABI compatibility mode. In this mode ODP APIs are never inlined. ABI
+compatibility ensures maximum application portability in cloud environments.
+
+`--enable-abi-compat=no`::
+Specify this option to enable the inlining of ODP APIs. This may result in
+improved performance at the cost of ABI compatibility and is suitable for
+applications running in embedded environments.
+
+Note that ODP applications retain source code portability between ODP
+implementations regardless of the ABI mode chosen. To move to a different ODP
+application running on a different ISA, code need simply be recompiled against
+that target ODP implementation.
+
+ SCTP Parsing Support
+The ODP classifier adds support for recognizing Stream Control Transmission
+Protocol (SCTP) packets. The APIs for this were previously not implemented.
+
+=== Packaging and Implementation Refinements
+
+ Remove dependency on Linux headers
+ODP no longer has a dependency on Linux headers. This will help make the
+odp-linux reference implementation more easily portable to non-Linux
+environments.
+
+ Remove dependency on helpers
+The odp-linux implementation has been made independent of the helper library
+to avoid circular dependency issues with packaging. Helper functions may use
+ODP APIs, however ODP implementations should not use helper functions.
+
+ Reorganization of `test` directory
+The `test` directory has been reorganized to better support a unified approach
+to ODP component testing. API tests now live in
+`test/common_plat/validation/api` instead of the former
+`test/validation`. With this change performance and validation tests, as well
+as common and platform-specific tests can all be part of a unified test
+hierarchy.
+
+The resulting test tree now looks like:
+
+.New `test` directory hierarchy
+-
+test
+├── common_plat
+│   ├── common
+│   ├── m4
+│   ├── miscellaneous
+│   ├── performance
+│   └── validation
+│   └── api
+│   ├── atomic
+│   ├── barrier
+│   ├── buffer
+│   ├── classification
+│   ├── cpumask
+│   ├── crypto
+│   ├── errno
+│   ├── hash
+│   ├── init
+│   ├── lock
+│   ├── packet
+│   ├── pktio
+│   ├── pool
+│   ├── queue
+│   ├── random
+│   ├── scheduler
+│   ├── shmem
+│   ├── std_clib
+│   ├── system
+│   ├── thread
+│   ├── time
+│   ├── timer
+│   └── traffic_mngr
+├── linux-generic
+│   ├── m4
+│   ├── mmap_vlan_ins
+│   ├── performance
+│   ├── pktio_ipc
+│   ├── ring
+│   └── validation
+│   └── api
+│   ├── pktio
+│   └── shmem
+└── m4
+-
+
+ Pools
+The maximum number of pools that may be created in the odp-linux reference
+implementation has been raised from 16 to 64.
+
+ Upgrade to DPDK 16.07
+The DPDK pktio support in odp-linux has been upgraded to work with DPDK 16.07.
+A number of miscellaneous fixes and performance improvements in this support
+are also present.
+
+ PktIO TAP Interface Classifier Support
+Packet I/O interfaces operating in TAP mode now can feed packets to the ODP
+classifier the same as other pktio modes can do.
+
+=== Performance Improvements
+
+ Burst-mode buffer allocation
+The scheduler and pktio components have been reworked to use burst-mode
+buffer allocation/deallocation, yielding a measurable performance gain in
+almost all cases.
+
+ Burst-mode queue operations
+ODP queues internally now attempt to use burst-mode enq/deq operations to
+accelerate performance where applicable.
+
+ Ring-based Scheduler Priority Queues
+The ODP scheduler has been enhanced to use ring-based priority queues, 
resulting
+in significantly better scalability in many core environments.
+
+ GitHub Automation Support
+ODP now supports the Travis framework needed to trigger CI automation in
+conjunction with GitHub. This support is considered experimental for now.

[lng-odp] [PATCHv6 3/3] configure.ac update version numbers

[Maxim Uvarov]

Default is abi compat mode, all interface functions changed,
so increase first number of .so

Signed-off-by: Maxim Uvarov 
---
 configure.ac | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/configure.ac b/configure.ac
index 7cb0c7a..de634dc 100644
--- a/configure.ac
+++ b/configure.ac
@@ -3,7 +3,7 @@ AC_PREREQ([2.5])
 # Set correct API version
 ##
 m4_define([odpapi_generation_version], [1])
-m4_define([odpapi_major_version], [11])
+m4_define([odpapi_major_version], [12])
 m4_define([odpapi_minor_version], [0])
 m4_define([odpapi_point_version], [0])
 m4_define([odpapi_version],
@@ -30,10 +30,10 @@ AM_SILENT_RULES([yes])
 ##
 # Set correct platform library version
 ##
-ODP_LIBSO_VERSION=111:0:0
+ODP_LIBSO_VERSION=112:0:0
 AC_SUBST(ODP_LIBSO_VERSION)
 
-ODPHELPER_LIBSO_VERSION=110:0:1
+ODPHELPER_LIBSO_VERSION=112:0:0
 AC_SUBST(ODPHELPER_LIBSO_VERSION)
 
 # Checks for programs.
-- 
2.7.1.250.gff4ea60

[lng-odp] [PATCHv6 1/3] configure.ac: disable shared library for non abi compat mode

[Maxim Uvarov]

original configure.ac enables abi compat mode by default,
even without --enable-abi-compat provided. And has broken
logic to disable abi compat mode. Correct logic to build abi
compat mode and option to disable it. Shared library is not
needed for non abi compat mode, so turn it off.

Signed-off-by: Maxim Uvarov 
---
 configure.ac | 9 ++---
 1 file changed, 6 insertions(+), 3 deletions(-)

diff --git a/configure.ac b/configure.ac
index be5a292..7cb0c7a 100644
--- a/configure.ac
+++ b/configure.ac
@@ -241,16 +241,19 @@ ODP_CFLAGS="$ODP_CFLAGS -DODP_DEBUG=$ODP_DEBUG"
 ODP_ABI_COMPAT=1
 abi_compat=yes
 AC_ARG_ENABLE([abi-compat],
-[  --enable-abi-compat build all targets in ABI compatible mode 
(default=yes)],
+[  --disable-abi-compatdisable build all targets in ABI compatible 
mode (default=no)],
 [if test "x$enableval" = "xyes"; then
ODP_ABI_COMPAT=1
-   abi_compat=yes
  else
ODP_ABI_COMPAT=0
-   abi_compat=no
 fi])
 AC_SUBST(ODP_ABI_COMPAT)
 
+if test $ODP_ABI_COMPAT -eq 0; then
+   enable_shared=no
+   abi_compat=no
+fi
+
 ##
 # Default warning setup
 ##
-- 
2.7.1.250.gff4ea60

[lng-odp] [PATCHv6 0/3] changelog: summary of changes for odp v1.12.0.0

[Maxim Uvarov]

v6: uh, finally we agreed with .so numberding.
added patch to disable shared library for non abi compat mode and corrected
description for configure option.

Bill Fischofer (1):
  changelog: summary of changes for odp v1.12.0.0

Maxim Uvarov (2):
  configure.ac: disable shared library for non abi compat mode
  configure.ac update version numbers

 CHANGELOG| 177 +++
 configure.ac |  15 +++--
 2 files changed, 186 insertions(+), 6 deletions(-)

-- 
2.7.1.250.gff4ea60

[lng-odp] [API-NEXT PATCHv9 15/15] doc: updating docs for the shm interface extension

[Christophe Milard]

Signed-off-by: Christophe Milard 
---
 doc/users-guide/users-guide.adoc | 68 ++--
 1 file changed, 66 insertions(+), 2 deletions(-)

diff --git a/doc/users-guide/users-guide.adoc b/doc/users-guide/users-guide.adoc
index 62f5833..078dd7c 100755
--- a/doc/users-guide/users-guide.adoc
+++ b/doc/users-guide/users-guide.adoc
@@ -649,13 +649,19 @@ mapping the shared memory block. There is no 
fragmentation.
 By default ODP threads are assumed to behave as cache coherent systems:
 Any change performed on a shared memory block is guaranteed to eventually
 become visible to other ODP threads sharing this memory block.
-(this behaviour may be altered by flags to `odp_shm_reserve()` in the future).
 Nevertheless, there is no implicit memory barrier associated with any action
 on shared memories: *When* a change performed by an ODP thread becomes visible
 to another ODP thread is not known: An application using shared memory
 blocks has to use some memory barrier provided by ODP to guarantee shared data
 validity between ODP threads.
 
+The virtual address at which a given memory block is mapped in different ODP
+threads may differ from ODP thread to ODP thread, if ODP threads have separate
+virtual spaces (for instance if ODP threads are implemented as processes).
+However, the ODP_SHM_SINGLE_VA flag can be used at `odp_shm_reserve()` time
+to guarantee address uniqueness in all ODP threads, regardless of their
+implementation or creation time.
+
 === Lookup by name
 As mentioned, shared memory handles can be sent from ODP threads to ODP
 threads using any IPC mechanism, and then the block address retrieved.
@@ -698,9 +704,49 @@ if (odp_shm_free(shm) != 0) {
 }
 
 
+=== sharing memory with the external world
+ODP provides ways of sharing memory with entities located outside
+ODP instances:
+
+Sharing a block of memory with an external (non ODP) thread is achieved
+by setting the ODP_SHM_PROC flag at `odp_shm_reserve()` time.
+How the memory block is retrieved on the Operating System side is
+implementation and Operating System dependent.
+
+Sharing a block of memory with an external ODP instance (running
+on the same Operating System) is achieved
+by setting the ODP_SHM_EXPORT flag at `odp_shm_reserve()` time.
+A block of memory created with this flag in an ODP instance A, can be "mapped"
+into a remote ODP instance B (on the same OS) by using the
+`odp_shm_import()`, on ODP instance B:
+
+.sharing memory between ODP instances: instance A
+[source,c]
+
+odp_shm_t shmA;
+shmA = odp_shm_reserve("memoryA", size, 0, ODP_SHM_EXPORT);
+
+
+.sharing memory between ODP instances: instance B
+[source,c]
+
+odp_shm_t shmB;
+odp_instance_t odpA;
+
+/* get ODP A instance handle by some OS method */
+odpA = ...
+
+/* get the shared memory exported by A:
+shmB = odp_shm_import("memoryA", odpA, "memoryB", 0, 0);
+
+
+Note that the handles shmA and shmB are scoped by each ODP instance
+(you can not use them outside the ODP instance they belong to).
+Also note that both ODP instances have to call `odp_shm_free()` when done.
+
 === Memory creation flags
 The last argument to odp_shm_reserve() is a set of ORed flags.
-Two flags are supported:
+The following flags are supported:
 
  ODP_SHM_PROC
 When this flag is given, the allocated shared memory will become visible
@@ -710,6 +756,12 @@ will be able to access the memory using native (non ODP) 
OS calls such as
 Each ODP implementation should provide a description on exactly how
 this mapping should be done on that specific platform.
 
+ ODP_SHM_EXPORT
+When this flag is given, the allocated shared memory will become visible
+to other ODP instances running on the same OS.
+Other ODP instances willing to see this exported memory should use the
+`odp_shm_import()` ODP function.
+
  ODP_SHM_SW_ONLY
 This flag tells ODP that the shared memory will be used by the ODP application
 software only: no HW (such as DMA, or other accelerator) will ever
@@ -719,6 +771,18 @@ implementation), except for `odp_shm_lookup()` and 
`odp_shm_free()`.
 ODP implementations may use this flag as a hint for performance optimization,
 or may as well ignore this flag.
 
+ ODP_SHM_SINGLE_VA
+This flag is used to guarantee the uniqueness of the address at which
+the shared memory is mapped: without this flag, a given memory block may be
+mapped at different virtual addresses (assuming the target have virtual
+addresses) by different ODP threads. This means that the value returned by
+`odp_shm_addr()` would be different in different threads, in this case.
+Setting this flag guarantees that all ODP threads sharing this memory
+block will see it at the same address (`odp_shm_addr()` would return the
+same value on all ODP threads, for a given memory block, in this case)
+Note that ODP implementations may have restrictions of the amount of memory
+which can be allocated with this flag.
+
 == Queues
 Queues are the 

[lng-odp] [API-NEXT PATCHv9 14/15] linux_gen: _ishm: decreasing the number of error messages when no huge pages

[Christophe Milard]

Signed-off-by: Christophe Milard 
---
 platform/linux-generic/_ishm.c | 22 --
 1 file changed, 16 insertions(+), 6 deletions(-)

diff --git a/platform/linux-generic/_ishm.c b/platform/linux-generic/_ishm.c
index b00f8f1..7d9c213 100644
--- a/platform/linux-generic/_ishm.c
+++ b/platform/linux-generic/_ishm.c
@@ -439,8 +439,12 @@ static int create_file(int block_index, huge_flag_t huge, 
uint64_t len,
 
fd = open(filename, oflag, S_IRUSR | S_IWUSR | S_IRGRP | S_IROTH);
if (fd < 0) {
-   ODP_ERR("open failed for %s: %s.\n",
-   filename, strerror(errno));
+   if (huge == HUGE)
+   ODP_DBG("open failed for %s: %s.\n",
+   filename, strerror(errno));
+   else
+   ODP_ERR("open failed for %s: %s.\n",
+   filename, strerror(errno));
return -1;
}
 
@@ -762,6 +766,7 @@ int _odp_ishm_reserve(const char *name, uint64_t size, int 
fd,
void *addr = NULL;/* mapping address */
int new_proc_entry;
struct stat statbuf;
+   static int  huge_error_printed;   /* to avoid millions of error...*/
 
odp_spinlock_lock(_tbl->lock);
 
@@ -836,11 +841,16 @@ int _odp_ishm_reserve(const char *name, uint64_t size, 
int fd,
len = (size + (page_hp_size - 1)) & (-page_hp_size);
addr = do_map(new_index, len, hp_align, flags, HUGE, );
 
-   if (addr == NULL)
-   ODP_DBG("No huge pages, fall back to normal pages, "
-   "check: /proc/sys/vm/nr_hugepages.\n");
-   else
+   if (addr == NULL) {
+   if (!huge_error_printed) {
+   ODP_ERR("No huge pages, fall back to normal "
+   "pages. "
+   "check: /proc/sys/vm/nr_hugepages.\n");
+   huge_error_printed = 1;
+   }
+   } else {
new_block->huge = HUGE;
+   }
}
 
/* Try normal pages if huge pages failed */
-- 
2.7.4

[lng-odp] [API-NEXT PATCHv9 13/15] linux-gen: _ishm: cleaning remaining block at odp_term_global

[Christophe Milard]

Remaining (forgotten, not freed) blocks are gathered and related files
cleaned when odp_term_global() is called. An error message is also issued
so the application writters get to know about these blocks

Signed-off-by: Christophe Milard 
---
 platform/linux-generic/_ishm.c | 13 +
 1 file changed, 13 insertions(+)

diff --git a/platform/linux-generic/_ishm.c b/platform/linux-generic/_ishm.c
index 7c373dc..b00f8f1 100644
--- a/platform/linux-generic/_ishm.c
+++ b/platform/linux-generic/_ishm.c
@@ -1505,12 +1505,25 @@ int _odp_ishm_term_local(void)
 int _odp_ishm_term_global(void)
 {
int ret = 0;
+   int index;
+   ishm_block_t *block;
 
if ((getpid() != odp_global_data.main_pid) ||
(syscall(SYS_gettid) != getpid()))
ODP_ERR("odp_term_global() must be performed by the main "
"ODP process!\n.");
 
+   /* cleanup possibly non freed memory (and complain a bit): */
+   for (index = 0; index < ISHM_MAX_NB_BLOCKS; index++) {
+   block = _tbl->block[index];
+   if (block->len != 0) {
+   ODP_ERR("block '%s' (file %s) was never freed "
+   "(cleaning up...).\n",
+   block->name, block->filename);
+   delete_file(block);
+   }
+   }
+
/* perform the last thread terminate which was postponed: */
ret = do_odp_ishm_term_local();
 
-- 
2.7.4

[lng-odp] [API-NEXT PATCHv9 12/15] test: linux-gen: api: shmem: test sharing memory between ODP instances

[Christophe Milard]

The platform tests odp/test/linux-generic/validation/api/shmem
are updated to both test ODP<->linux process memory sharing, but also test
ODP to ODP (different instances) memory sharing.
shmem_linux is the main test process, and shmem_linux.c contains (at
file top) a chart flow of the test procedure.

Signed-off-by: Christophe Milard 
---
 test/linux-generic/validation/api/shmem/.gitignore |   3 +-
 .../linux-generic/validation/api/shmem/Makefile.am |  22 ++--
 .../validation/api/shmem/shmem_linux.c | 140 +++--
 .../api/shmem/{shmem_odp.c => shmem_odp1.c}|  10 +-
 .../api/shmem/{shmem_odp.h => shmem_odp1.h}|   0
 .../validation/api/shmem/shmem_odp2.c  |  95 ++
 .../validation/api/shmem/shmem_odp2.h  |   7 ++
 7 files changed, 230 insertions(+), 47 deletions(-)
 rename test/linux-generic/validation/api/shmem/{shmem_odp.c => shmem_odp1.c} 
(81%)
 rename test/linux-generic/validation/api/shmem/{shmem_odp.h => shmem_odp1.h} 
(100%)
 create mode 100644 test/linux-generic/validation/api/shmem/shmem_odp2.c
 create mode 100644 test/linux-generic/validation/api/shmem/shmem_odp2.h

diff --git a/test/linux-generic/validation/api/shmem/.gitignore 
b/test/linux-generic/validation/api/shmem/.gitignore
index 7627079..74195f5 100644
--- a/test/linux-generic/validation/api/shmem/.gitignore
+++ b/test/linux-generic/validation/api/shmem/.gitignore
@@ -1,2 +1,3 @@
 shmem_linux
-shmem_odp
+shmem_odp1
+shmem_odp2
diff --git a/test/linux-generic/validation/api/shmem/Makefile.am 
b/test/linux-generic/validation/api/shmem/Makefile.am
index 341747f..b0ae627 100644
--- a/test/linux-generic/validation/api/shmem/Makefile.am
+++ b/test/linux-generic/validation/api/shmem/Makefile.am
@@ -2,19 +2,27 @@ include ../Makefile.inc
 
 #the main test program is shmem_linux, which, in turn, starts a shmem_odp:
 test_PROGRAMS = shmem_linux$(EXEEXT)
-test_extra_PROGRAMS = shmem_odp$(EXEEXT)
+test_extra_PROGRAMS = shmem_odp1$(EXEEXT) shmem_odp2$(EXEEXT)
 test_extradir = $(testdir)
 
 #shmem_linux is stand alone, pure linux (no ODP):
 dist_shmem_linux_SOURCES = shmem_linux.c
 shmem_linux_LDFLAGS = $(AM_LDFLAGS) -lrt
 
-#shmem_odp is the odp part:
-dist_shmem_odp_SOURCES = shmem_odp.c
-shmem_odp_CFLAGS = $(AM_CFLAGS) \
+#shmem_odp1 and shmem_odp2 are the 2 ODP processes:
+dist_shmem_odp1_SOURCES = shmem_odp1.c
+shmem_odp1_CFLAGS = $(AM_CFLAGS) \
   $(INCCUNIT_COMMON) \
   $(INCODP)
-shmem_odp_LDFLAGS = $(AM_LDFLAGS)
-shmem_odp_LDADD = $(LIBCUNIT_COMMON) $(LIBODP)
+shmem_odp1_LDFLAGS = $(AM_LDFLAGS)
+shmem_odp1_LDADD = $(LIBCUNIT_COMMON) $(LIBODP)
 
-noinst_HEADERS = shmem_common.h shmem_linux.h shmem_odp.h
+dist_shmem_odp2_SOURCES = shmem_odp2.c
+shmem_odp2_CFLAGS = $(AM_CFLAGS) \
+  $(INCCUNIT_COMMON) \
+  $(INCODP)
+shmem_odp2_LDFLAGS = $(AM_LDFLAGS)
+shmem_odp2_LDADD = $(LIBCUNIT_COMMON) $(LIBODP)
+
+
+noinst_HEADERS = shmem_common.h shmem_linux.h shmem_odp1.h shmem_odp2.h
diff --git a/test/linux-generic/validation/api/shmem/shmem_linux.c 
b/test/linux-generic/validation/api/shmem/shmem_linux.c
index 9ab0e2b..39473f3 100644
--- a/test/linux-generic/validation/api/shmem/shmem_linux.c
+++ b/test/linux-generic/validation/api/shmem/shmem_linux.c
@@ -5,8 +5,10 @@
  */
 
 /* this test makes sure that odp shared memory created with the ODP_SHM_PROC
- * flag is visible under linux. It therefore checks both that the device
- * name under /dev/shm is correct, and also checks that the memory contents
+ * flag is visible under linux, and checks that memory created with the
+ * ODP_SHM_EXPORT flag is visible by other ODP instances.
+ * It therefore checks both that the link
+ * name under /tmp is correct, and also checks that the memory contents
  * is indeed shared.
  * we want:
  * -the odp test to run using C UNIT
@@ -15,18 +17,47 @@
  *
  * To achieve this, the flow of operations is as follows:
  *
- *   linux process (main, non odp) |   ODP process
- *   (shmem_linux.c)   |   (shmem_odp.c)
+ *   linux process (main, non odp) |
+ *   (shmem_linux.c)   |
+ * |
+ * |
  * |
  *   main()|
- *   forks odp process |  allocate shmem
- *   wait for named pipe creation  |  populate shmem
+ *   forks odp_app1 process|
+ *   wait for named pipe creation  |
+ * |
+ * |   ODP_APP1 process
+ * |   (shmem_odp1.c)
+ * |
+ * |  allocate shmem
+ * |  populate shmem
  * |  create named pipe
- *   read shared memory

[lng-odp] [API-NEXT PATCHv9 11/15] linux-gen: shm: add flag and function to share memory between ODP instances

[Christophe Milard]

Implemented by calling the related functions from _ishm.

Signed-off-by: Christophe Milard 
---
 platform/linux-generic/odp_shared_memory.c | 31 +++---
 1 file changed, 28 insertions(+), 3 deletions(-)

diff --git a/platform/linux-generic/odp_shared_memory.c 
b/platform/linux-generic/odp_shared_memory.c
index 2377f16..d2bb74c 100644
--- a/platform/linux-generic/odp_shared_memory.c
+++ b/platform/linux-generic/odp_shared_memory.c
@@ -24,6 +24,21 @@ static inline odp_shm_t to_handle(uint32_t index)
return _odp_cast_scalar(odp_shm_t, index + 1);
 }
 
+static uint32_t get_ishm_flags(uint32_t flags)
+{
+   uint32_t f = 0; /* internal ishm flags */
+
+   /* set internal ishm flags according to API flags:
+* note that both ODP_SHM_PROC and ODP_SHM_EXPORT maps to
+* _ODP_ISHM_LINK as in the linux-gen implementation there is
+* no difference between exporting to another ODP instance or
+* another linux process */
+   f |= (flags & (ODP_SHM_PROC | ODP_SHM_EXPORT)) ? _ODP_ISHM_EXPORT : 0;
+   f |= (flags & ODP_SHM_SINGLE_VA) ? _ODP_ISHM_SINGLE_VA : 0;
+
+   return f;
+}
+
 int odp_shm_capability(odp_shm_capability_t *capa)
 {
memset(capa, 0, sizeof(odp_shm_capability_t));
@@ -41,9 +56,7 @@ odp_shm_t odp_shm_reserve(const char *name, uint64_t size, 
uint64_t align,
int block_index;
int flgs = 0; /* internal ishm flags */
 
-   /* set internal ishm flags according to API flags: */
-   flgs |= (flags & ODP_SHM_PROC) ? _ODP_ISHM_EXPORT : 0;
-   flgs |= (flags & ODP_SHM_SINGLE_VA) ? _ODP_ISHM_SINGLE_VA : 0;
+   flgs = get_ishm_flags(flags);
 
/* all mem reserved through this interface is requested to be locked: */
flgs |= (flags & _ODP_ISHM_LOCK);
@@ -55,6 +68,18 @@ odp_shm_t odp_shm_reserve(const char *name, uint64_t size, 
uint64_t align,
return ODP_SHM_INVALID;
 }
 
+odp_shm_t odp_shm_import(const char *remote_name,
+odp_instance_t odp_inst,
+const char *local_name)
+{
+   int ret;
+
+   ret =  _odp_ishm_find_exported(remote_name, (pid_t)odp_inst,
+  local_name);
+
+   return to_handle(ret);
+}
+
 int odp_shm_free(odp_shm_t shm)
 {
return _odp_ishm_free_by_index(from_handle(shm));
-- 
2.7.4

[lng-odp] [API-NEXT PATCHv9 09/15] test: api: shmem: new proper tests for shm API

[Christophe Milard]

The shmem "sunnydays" tests for the north interface API are replaced with
proper tests, testing memory allocation at different time (before and
after ODP thread creation, i.e. the tests make sure shmem behaves
the same regardless of fork time). The tests also include stress testing
trying to provoque race conditions. The new shmem tests do not assume
pthreads any longer and are runnable in process mode.

Signed-off-by: Christophe Milard 
---
 test/common_plat/validation/api/shmem/shmem.c | 687 --
 test/common_plat/validation/api/shmem/shmem.h |   5 +-
 2 files changed, 658 insertions(+), 34 deletions(-)

diff --git a/test/common_plat/validation/api/shmem/shmem.c 
b/test/common_plat/validation/api/shmem/shmem.c
index cbff673..6ea92d9 100644
--- a/test/common_plat/validation/api/shmem/shmem.c
+++ b/test/common_plat/validation/api/shmem/shmem.c
@@ -7,82 +7,703 @@
 #include 
 #include 
 #include "shmem.h"
+#include 
 
-#define ALIGE_SIZE  (128)
-#define TESTNAME "cunit_test_shared_data"
+#define ALIGN_SIZE  (128)
+#define MEM_NAME "test_shmem"
+#define NAME_LEN (sizeof(MEM_NAME) + 20)
 #define TEST_SHARE_FOO (0xf0f0f0f0)
 #define TEST_SHARE_BAR (0xf0f0f0f)
+#define SMALL_MEM 10
+#define MEDIUM_MEM 4096
+#define BIG_MEM 65536
+#define STRESS_SIZE 32 /* power of 2 and <=256 */
+#define STRESS_RANDOM_SZ 5
+#define STRESS_ITERATION 5000
 
-static odp_barrier_t test_barrier;
+typedef enum {
+   STRESS_FREE, /* entry is free and can be allocated */
+   STRESS_BUSY, /* entry is being processed: don't touch */
+   STRESS_ALLOC /* entry is allocated  and can be freed */
+} stress_state_t;
 
-static int run_shm_thread(void *arg ODP_UNUSED)
+typedef struct {
+   stress_state_t state;
+   odp_shm_t shm;
+   char name[NAME_LEN];
+   void *address;
+   uint32_t flags;
+   uint32_t size;
+   uint64_t align;
+   uint8_t data_val;
+} stress_data_t;
+
+typedef struct {
+   odp_barrier_t test_barrier1;
+   odp_barrier_t test_barrier2;
+   odp_barrier_t test_barrier3;
+   odp_barrier_t test_barrier4;
+   uint32_t foo;
+   uint32_t bar;
+   odp_atomic_u32_t index;
+   uint32_t nb_threads;
+   odp_shm_t shm[MAX_WORKERS];
+   void *address[MAX_WORKERS];
+   char name[MAX_WORKERS][NAME_LEN];
+   odp_spinlock_t  stress_lock;
+   stress_data_t stress[STRESS_SIZE];
+} shared_test_data_t;
+
+/* memory stuff expected to fit in a single page */
+typedef struct {
+   int data[SMALL_MEM];
+} shared_test_data_small_t;
+
+/* memory stuff expected to fit in a huge page */
+typedef struct {
+   int data[MEDIUM_MEM];
+} shared_test_data_medium_t;
+
+/* memory stuff expected to fit in many huge pages */
+typedef struct {
+   int data[BIG_MEM];
+} shared_test_data_big_t;
+
+/*
+ * thread part for the shmem_test_basic test
+ */
+static int run_test_basic_thread(void *arg ODP_UNUSED)
 {
odp_shm_info_t  info;
odp_shm_t shm;
-   test_shared_data_t *test_shared_data;
+   shared_test_data_t *shared_test_data;
int thr;
 
-   odp_barrier_wait(_barrier);
thr = odp_thread_id();
printf("Thread %i starts\n", thr);
 
-   shm = odp_shm_lookup(TESTNAME);
+   shm = odp_shm_lookup(MEM_NAME);
CU_ASSERT(ODP_SHM_INVALID != shm);
-   test_shared_data = odp_shm_addr(shm);
-   CU_ASSERT(TEST_SHARE_FOO == test_shared_data->foo);
-   CU_ASSERT(TEST_SHARE_BAR == test_shared_data->bar);
+   shared_test_data = odp_shm_addr(shm);
+   CU_ASSERT(NULL != shared_test_data);
+
+   odp_barrier_wait(_test_data->test_barrier1);
+   odp_shm_print_all();
+   CU_ASSERT(TEST_SHARE_FOO == shared_test_data->foo);
+   CU_ASSERT(TEST_SHARE_BAR == shared_test_data->bar);
CU_ASSERT(0 == odp_shm_info(shm, ));
-   CU_ASSERT(0 == strcmp(TESTNAME, info.name));
+   CU_ASSERT(0 == strcmp(MEM_NAME, info.name));
CU_ASSERT(0 == info.flags);
-   CU_ASSERT(test_shared_data == info.addr);
-   CU_ASSERT(sizeof(test_shared_data_t) <= info.size);
-#ifdef MAP_HUGETLB
-   CU_ASSERT(odp_sys_huge_page_size() == info.page_size);
-#else
-   CU_ASSERT(odp_sys_page_size() == info.page_size);
-#endif
+   CU_ASSERT(shared_test_data == info.addr);
+   CU_ASSERT(sizeof(shared_test_data_t) <= info.size);
+   CU_ASSERT((info.page_size == odp_sys_huge_page_size()) ||
+ (info.page_size == odp_sys_page_size()))
odp_shm_print_all();
 
fflush(stdout);
return CU_get_number_of_failures();
 }
 
-void shmem_test_odp_shm_sunnyday(void)
+/*
+ * test basic things: shmem creation, info, share, and free
+ */
+void shmem_test_basic(void)
 {
pthrd_arg thrdarg;
odp_shm_t shm;
-   test_shared_data_t *test_shared_data;
+   shared_test_data_t *shared_test_data;
odp_cpumask_t unused;
 
-   shm = odp_shm_reserve(TESTNAME,
-   

[lng-odp] [API-NEXT PATCHv9 10/15] linux-gen: _ishm: adding function to map memory from other ODP

[Christophe Milard]

functionality to export and map memory between ODP instance is added:
This includes:
- a bit of simplification in _odp_ishm_reserve() for externaly provided
  file descriptors.
- a new function, _odp_ishm_find_exported() to map memory from
  other ODP instances (On same OS)

Signed-off-by: Christophe Milard 
---
 platform/linux-generic/_ishm.c  | 160 +++-
 platform/linux-generic/include/_ishm_internal.h |   5 +
 2 files changed, 136 insertions(+), 29 deletions(-)

diff --git a/platform/linux-generic/_ishm.c b/platform/linux-generic/_ishm.c
index eb94306..7c373dc 100644
--- a/platform/linux-generic/_ishm.c
+++ b/platform/linux-generic/_ishm.c
@@ -152,6 +152,7 @@ typedef struct ishm_fragment {
  * will allocate both a block and a fragment.
  * Blocks contain only global data common to all processes.
  */
+typedef enum {UNKNOWN, HUGE, NORMAL, EXTERNAL} huge_flag_t;
 typedef struct ishm_block {
char name[ISHM_NAME_MAXLEN];/* name for the ishm block (if any) */
char filename[ISHM_FILENAME_MAXLEN]; /* name of the .../odp-* file  */
@@ -162,7 +163,7 @@ typedef struct ishm_block {
void *start; /* only valid if _ODP_ISHM_SINGLE_VA is set*/
uint64_t len;/* length. multiple of page size. 0 if free*/
ishm_fragment_t *fragment; /* used when _ODP_ISHM_SINGLE_VA is used */
-   int   huge; /* true if this segment is mapped using huge pages  */
+   huge_flag_t huge;/* page type: external means unknown here. */
uint64_t seq;   /* sequence number, incremented on alloc and free   */
uint64_t refcnt;/* number of linux processes mapping this block */
 } ishm_block_t;
@@ -400,7 +401,7 @@ static void free_fragment(ishm_fragment_t *fragmnt)
  * or /mnt/huge/odp-- (for huge pages)
  * Return the new file descriptor, or -1 on error.
  */
-static int create_file(int block_index, int huge, uint64_t len,
+static int create_file(int block_index, huge_flag_t huge, uint64_t len,
   uint32_t flags, uint32_t align)
 {
char *name;
@@ -419,10 +420,11 @@ static int create_file(int block_index, int huge, 
uint64_t len,
 ishm_tbl->dev_seq++);
 
/* huge dir must be known to create files there!: */
-   if (huge && !odp_global_data.hugepage_info.default_huge_page_dir)
+   if ((huge == HUGE) &&
+   (!odp_global_data.hugepage_info.default_huge_page_dir))
return -1;
 
-   if (huge)
+   if (huge == HUGE)
snprintf(filename, ISHM_FILENAME_MAXLEN,
 ISHM_FILENAME_FORMAT,
 odp_global_data.hugepage_info.default_huge_page_dir,
@@ -502,7 +504,7 @@ static void delete_file(ishm_block_t *block)
  * Mutex must be assured by the caller.
  */
 static void *do_map(int block_index, uint64_t len, uint32_t align,
-   uint32_t flags, int huge, int *fd)
+   uint32_t flags, huge_flag_t huge, int *fd)
 {
ishm_block_t *new_block;  /* entry in the main block table   */
void *addr = NULL;
@@ -552,8 +554,6 @@ static void *do_map(int block_index, uint64_t len, uint32_t 
align,
return NULL;
}
 
-   new_block->huge = huge;
-
return mapped_addr;
 }
 
@@ -756,27 +756,21 @@ int _odp_ishm_reserve(const char *name, uint64_t size, 
int fd,
int new_index;/* index in the main block table*/
ishm_block_t *new_block;  /* entry in the main block table*/
uint64_t page_sz; /* normal page size. usually 4K*/
-   uint64_t alloc_size;  /* includes extra for alignement*/
uint64_t page_hp_size;/* huge page size */
-   uint64_t alloc_hp_size;   /* includes extra for alignement*/
uint32_t hp_align;
uint64_t len; /* mapped length */
void *addr = NULL;/* mapping address */
int new_proc_entry;
-
-   page_sz = odp_sys_page_size();
+   struct stat statbuf;
 
odp_spinlock_lock(_tbl->lock);
 
/* update this process view... */
procsync();
 
-   /* roundup to page size */
-   alloc_size = (size + (page_sz - 1)) & (-page_sz);
-
+   /* Get system page sizes: page_hp_size is 0 if no huge page available*/
+   page_sz  = odp_sys_page_size();
page_hp_size = odp_sys_huge_page_size();
-   /* roundup to page size */
-   alloc_hp_size = (size + (page_hp_size - 1)) & (-page_hp_size);
 
/* check if name already exists */
if (name && (find_block_by_name(name) >= 0)) {
@@ -809,8 +803,24 @@ int _odp_ishm_reserve(const char *name, uint64_t size, int 
fd,
else
new_block->name[0] = 0;
 
-   /* Try first huge pages when possible and needed: */
-   if (page_hp_size && (alloc_size > 

[lng-odp] [API-NEXT PATCHv9 07/15] api: shm: add flags to shm_reserve and function to find external mem

[Christophe Milard]

The ODP_SHM_SINGLE_VA flag is created: when set (at odp_shm_reserve()),
this flag guarantees that all ODP threads sharing this memory
block will see the block at the same address (regadless of ODP
thread type -pthread vs process- or fork time)

The flag ODP_SHM_EXPORT is added: when passed at odp_shm_reserve() time
the memory block becomes visible to other ODP instances.
The function odp_shm_import() is added: this function enables to
reserve block of memories exported by other ODP instances (using the
ODP_SHM_EXPORT flag).

Signed-off-by: Christophe Milard 
---
 include/odp/api/spec/shared_memory.h | 46 
 1 file changed, 41 insertions(+), 5 deletions(-)

diff --git a/include/odp/api/spec/shared_memory.h 
b/include/odp/api/spec/shared_memory.h
index 8c76807..885751d 100644
--- a/include/odp/api/spec/shared_memory.h
+++ b/include/odp/api/spec/shared_memory.h
@@ -14,6 +14,7 @@
 #ifndef ODP_API_SHARED_MEMORY_H_
 #define ODP_API_SHARED_MEMORY_H_
 #include 
+#include 
 
 #ifdef __cplusplus
 extern "C" {
@@ -43,12 +44,25 @@ extern "C" {
 #define ODP_SHM_NAME_LEN 32
 
 /*
- * Shared memory flags
+ * Shared memory flags:
  */
-
-/* Share level */
-#define ODP_SHM_SW_ONLY 0x1 /**< Application SW only, no HW access */
-#define ODP_SHM_PROC0x2 /**< Share with external processes */
+#define ODP_SHM_SW_ONLY0x1 /**< Application SW only, no HW 
access   */
+#define ODP_SHM_PROC   0x2 /**< Share with external processes   */
+/**
+ * Single virtual address
+ *
+ * When set, this flag guarantees that all ODP threads sharing this
+ * memory block will see the block at the same address - regardless
+ * of ODP thread type (e.g. pthread vs. process (or fork process time)).
+ */
+#define ODP_SHM_SINGLE_VA  0x4
+/**
+ * Export memory
+ *
+ * When set, the memory block becomes visible to other ODP instances
+ * through odp_shm_import().
+ */
+#define ODP_SHM_EXPORT 0x08
 
 /**
  * Shared memory block info
@@ -135,6 +149,28 @@ int odp_shm_free(odp_shm_t shm);
  */
 odp_shm_t odp_shm_lookup(const char *name);
 
+/**
+ * Import a block of shared memory, exported by another ODP instance
+ *
+ * This call creates a new handle for accessing a shared memory block created
+ * (with ODP_SHM_EXPORT flag) by another ODP instance. An instance may have
+ * only a single handle to the same block. Application must not access the
+ * block after freeing the handle. When an imported handle is freed, only
+ * the calling instance is affected. The exported block may be freed only
+ * after all other instances have stopped accessing the block.
+ *
+ * @param remote_name  Name of the block, in the remote ODP instance
+ * @param odp_inst Remote ODP instance, as returned by odp_init_global()
+ * @param local_name   Name given to the block, in the local ODP instance
+ *May be NULL, if the application doesn't need a name
+ *(for a lookup).
+ *
+ * @return A handle to access a block exported by another ODP instance.
+ * @retval ODP_SHM_INVALID on failure
+ */
+odp_shm_t odp_shm_import(const char *remote_name,
+odp_instance_t odp_inst,
+const char *local_name);
 
 /**
  * Shared memory block address
-- 
2.7.4

[lng-odp] [API-NEXT PATCHv9 08/15] linux-gen: shm: new ODP_SHM_SINGLE_VA flag implementation

[Christophe Milard]

This flag guarentess the unicity the the block address on all ODP threads.
The patch just exposes the _ODP_ISHM_SINGLE_VA flag of the internal memory
allocator, ishm.

Signed-off-by: Christophe Milard 
---
 platform/linux-generic/odp_shared_memory.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/platform/linux-generic/odp_shared_memory.c 
b/platform/linux-generic/odp_shared_memory.c
index 9e916e9..2377f16 100644
--- a/platform/linux-generic/odp_shared_memory.c
+++ b/platform/linux-generic/odp_shared_memory.c
@@ -43,6 +43,7 @@ odp_shm_t odp_shm_reserve(const char *name, uint64_t size, 
uint64_t align,
 
/* set internal ishm flags according to API flags: */
flgs |= (flags & ODP_SHM_PROC) ? _ODP_ISHM_EXPORT : 0;
+   flgs |= (flags & ODP_SHM_SINGLE_VA) ? _ODP_ISHM_SINGLE_VA : 0;
 
/* all mem reserved through this interface is requested to be locked: */
flgs |= (flags & _ODP_ISHM_LOCK);
-- 
2.7.4

[lng-odp] [API-NEXT PATCHv9 06/15] linux-gen: push internal flag definition

[Christophe Milard]

File platform/linux-generic/include/odp_shm_internal.h exposes shm
internals used by IPC. The bits used by the internal flags are moved
to make room for more "official" values.
The platform/linux-generic/include/odp_shm_internal.h file should really
be removed when _ishm is used, but as long as we have the current IPC,
removing the file would break compilation.

Signed-off-by: Christophe Milard 
---
 platform/linux-generic/include/odp_shm_internal.h | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/platform/linux-generic/include/odp_shm_internal.h 
b/platform/linux-generic/include/odp_shm_internal.h
index 30e60f7..8bd105d 100644
--- a/platform/linux-generic/include/odp_shm_internal.h
+++ b/platform/linux-generic/include/odp_shm_internal.h
@@ -16,8 +16,8 @@ extern "C" {
 #define SHM_DEVNAME_MAXLEN (ODP_SHM_NAME_LEN + 16)
 #define SHM_DEVNAME_FORMAT "/odp-%d-%s" /* /dev/shm/odp-- */
 
-#define _ODP_SHM_PROC_NOCREAT 0x4  /**< Do not create shm if not exist */
-#define _ODP_SHM_O_EXCL  0x8  /**< Do not create shm if exist */
+#define _ODP_SHM_PROC_NOCREAT 0x40  /**< Do not create shm if not exist */
+#define _ODP_SHM_O_EXCL  0x80  /**< Do not create shm if exist */
 
 #ifdef __cplusplus
 }
-- 
2.7.4

[lng-odp] [API-NEXT PATCHv9 05/15] linux-gen: use ishm as north API mem allocator

[Christophe Milard]

The odp shared_memory API is changed to use the ODP internal memory
allocator: _ishm.
_ishm supports memory sharing between processes, regardless of fork time.
The test testing the ODP_SHM_PROC flag is also changed to cope with the
new OS sharing interface used by _ishm (link in /tmp).

Signed-off-by: Christophe Milard 
---
 platform/linux-generic/include/odp_internal.h  |   5 -
 platform/linux-generic/odp_init.c  |  19 -
 platform/linux-generic/odp_shared_memory.c | 406 ++---
 .../validation/api/shmem/shmem_linux.c | 100 -
 4 files changed, 111 insertions(+), 419 deletions(-)

diff --git a/platform/linux-generic/include/odp_internal.h 
b/platform/linux-generic/include/odp_internal.h
index 2f29d3d..6a80d9d 100644
--- a/platform/linux-generic/include/odp_internal.h
+++ b/platform/linux-generic/include/odp_internal.h
@@ -60,7 +60,6 @@ enum init_stage {
SYSINFO_INIT,
FDSERVER_INIT,
ISHM_INIT,
-   SHM_INIT,
THREAD_INIT,
POOL_INIT,
QUEUE_INIT,
@@ -90,10 +89,6 @@ int odp_thread_init_local(odp_thread_type_t type);
 int odp_thread_term_local(void);
 int odp_thread_term_global(void);
 
-int odp_shm_init_global(void);
-int odp_shm_term_global(void);
-int odp_shm_init_local(void);
-
 int odp_pool_init_global(void);
 int odp_pool_init_local(void);
 int odp_pool_term_global(void);
diff --git a/platform/linux-generic/odp_init.c 
b/platform/linux-generic/odp_init.c
index 6e58ab6..d40a83c 100644
--- a/platform/linux-generic/odp_init.c
+++ b/platform/linux-generic/odp_init.c
@@ -123,12 +123,6 @@ int odp_init_global(odp_instance_t *instance,
}
stage = ISHM_INIT;
 
-   if (odp_shm_init_global()) {
-   ODP_ERR("ODP shm init failed.\n");
-   goto init_failed;
-   }
-   stage = SHM_INIT;
-
if (odp_thread_init_global()) {
ODP_ERR("ODP thread init failed.\n");
goto init_failed;
@@ -282,13 +276,6 @@ int _odp_term_global(enum init_stage stage)
}
/* Fall through */
 
-   case SHM_INIT:
-   if (odp_shm_term_global()) {
-   ODP_ERR("ODP shm term failed.\n");
-   rc = -1;
-   }
-   /* Fall through */
-
case ISHM_INIT:
if (_odp_ishm_term_global()) {
ODP_ERR("ODP ishm term failed.\n");
@@ -346,12 +333,6 @@ int odp_init_local(odp_instance_t instance, 
odp_thread_type_t thr_type)
}
stage = ISHM_INIT;
 
-   if (odp_shm_init_local()) {
-   ODP_ERR("ODP shm local init failed.\n");
-   goto init_fail;
-   }
-   stage = SHM_INIT;
-
if (odp_thread_init_local(thr_type)) {
ODP_ERR("ODP thread local init failed.\n");
goto init_fail;
diff --git a/platform/linux-generic/odp_shared_memory.c 
b/platform/linux-generic/odp_shared_memory.c
index 550af27..9e916e9 100644
--- a/platform/linux-generic/odp_shared_memory.c
+++ b/platform/linux-generic/odp_shared_memory.c
@@ -4,434 +4,88 @@
  * SPDX-License-Identifier: BSD-3-Clause
  */
 
-#include 
-
-#include 
-#include 
-#include 
-#include 
-#include 
-#include 
-#include 
-#include 
-#include 
 #include 
-
-#include 
-#include 
-#include 
-#include 
-#include 
-
-#include 
+#include 
+#include 
+#include 
+#include <_ishm_internal.h>
 #include 
-#include 
-#include 
 
 ODP_STATIC_ASSERT(ODP_CONFIG_SHM_BLOCKS >= ODP_CONFIG_POOLS,
  "ODP_CONFIG_SHM_BLOCKS < ODP_CONFIG_POOLS");
 
-typedef struct {
-   char  name[ODP_SHM_NAME_LEN];
-   uint64_t  size;
-   uint64_t  align;
-   uint64_t  alloc_size;
-   void  *addr_orig;
-   void  *addr;
-   int   huge;
-   odp_shm_t hdl;
-   uint32_t  flags;
-   uint64_t  page_sz;
-   int   fd;
-
-} odp_shm_block_t;
-
-
-typedef struct {
-   odp_shm_block_t block[ODP_CONFIG_SHM_BLOCKS];
-   odp_spinlock_t  lock;
-
-} odp_shm_table_t;
-
-
-#ifndef MAP_ANONYMOUS
-#define MAP_ANONYMOUS MAP_ANON
-#endif
-
-
-/* Global shared memory table */
-static odp_shm_table_t *odp_shm_tbl;
-
-
 static inline uint32_t from_handle(odp_shm_t shm)
 {
return _odp_typeval(shm) - 1;
 }
 
-
 static inline odp_shm_t to_handle(uint32_t index)
 {
return _odp_cast_scalar(odp_shm_t, index + 1);
 }
 
-
-int odp_shm_init_global(void)
-{
-   void *addr;
-
-#ifndef MAP_HUGETLB
-   ODP_DBG("NOTE: mmap does not support huge pages\n");
-#endif
-
-   addr = mmap(NULL, sizeof(odp_shm_table_t),
-   PROT_READ | PROT_WRITE, MAP_SHARED | MAP_ANONYMOUS, -1, 0);
-
-   if (addr == MAP_FAILED)
-   return -1;
-
-   odp_shm_tbl = addr;
-
-   memset(odp_shm_tbl, 0, sizeof(odp_shm_table_t));
-   odp_spinlock_init(_shm_tbl->lock);
-
-   return 0;
-}
-
-int odp_shm_term_global(void)
-{
- 

[lng-odp] [API-NEXT PATCHv9 04/15] linux-gen: _ishm: allow memory alloc/free at global init/term

[Christophe Milard]

_ishm.c assumed that both _ishm_init_global() and _ishm_init_local()
had been run to work properly. This assumption turns out the be a problem
if _ishm is to be used as main memory allocator, as many modules
init_global functions assume the availability of the odp_reserve()
function before any init_local function is called.
Likewise, many term_global() functions assume the availability
of the odp_shm_free() function after all odp_term_local() have run.
This patch runs _ishm_init_local() in advance for the main ODP thread
and postpones the execution of_ishm_term_local() to init_global()
time for the main process, hence making the ishm_reserve()
and ishm_free() functions available at init_global/term_global time.

Signed-off-by: Christophe Milard 
---
 platform/linux-generic/_ishm.c | 189 +
 1 file changed, 118 insertions(+), 71 deletions(-)

diff --git a/platform/linux-generic/_ishm.c b/platform/linux-generic/_ishm.c
index e3ab8b4..eb94306 100644
--- a/platform/linux-generic/_ishm.c
+++ b/platform/linux-generic/_ishm.c
@@ -156,7 +156,6 @@ typedef struct ishm_block {
char name[ISHM_NAME_MAXLEN];/* name for the ishm block (if any) */
char filename[ISHM_FILENAME_MAXLEN]; /* name of the .../odp-* file  */
char exptname[ISHM_FILENAME_MAXLEN]; /* name of the export file */
-   int  main_odpthread; /* The thread which did the initial reserve*/
uint32_t user_flags; /* any flags the user want to remember.*/
uint32_t flags;  /* block creation flags.   */
uint64_t user_len;   /* length, as requested at reserve time.   */
@@ -178,6 +177,7 @@ typedef struct ishm_block {
 typedef struct {
odp_spinlock_t  lock;
uint64_t dev_seq;   /* used when creating device names */
+   uint32_t odpthread_cnt; /* number of running ODP threads   */
ishm_block_t  block[ISHM_MAX_NB_BLOCKS];
 } ishm_table_t;
 static ishm_table_t *ishm_tbl;
@@ -864,7 +864,6 @@ int _odp_ishm_reserve(const char *name, uint64_t size, int 
fd,
new_block->user_flags = user_flags;
new_block->seq++;
new_block->refcnt = 1;
-   new_block->main_odpthread = odp_thread_id();
new_block->start = addr; /* only for SINGLE_VA*/
 
/* the allocation succeeded: update the process local view */
@@ -907,10 +906,8 @@ static int block_free(int block_index)
 
proc_index = procfind_block(block_index);
if (proc_index >= 0) {
-   /* close the fd, unless if it was externaly provided */
-   if ((block->filename[0] != 0) ||
-   (odp_thread_id() != block->main_odpthread))
-   close(ishm_proctable->entry[proc_index].fd);
+   /* close the related fd */
+   close(ishm_proctable->entry[proc_index].fd);
 
/* remove the mapping and possible fragment */
do_unmap(ishm_proctable->entry[proc_index].start,
@@ -1201,12 +1198,62 @@ int _odp_ishm_info(int block_index, _odp_ishm_info_t 
*info)
return 0;
 }
 
+static int do_odp_ishm_init_local(void)
+{
+   int i;
+   int block_index;
+
+   /*
+* the ishm_process table is local to each linux process
+* Check that no other linux threads (of same or ancestor processes)
+* have already created the table, and create it if needed.
+* We protect this with the general ishm lock to avoid
+* init race condition of different running threads.
+*/
+   odp_spinlock_lock(_tbl->lock);
+   ishm_tbl->odpthread_cnt++; /* count ODPthread (pthread or process) */
+   if (!ishm_proctable) {
+   ishm_proctable = malloc(sizeof(ishm_proctable_t));
+   if (!ishm_proctable) {
+   odp_spinlock_unlock(_tbl->lock);
+   return -1;
+   }
+   memset(ishm_proctable, 0, sizeof(ishm_proctable_t));
+   }
+   if (syscall(SYS_gettid) != getpid())
+   ishm_proctable->thrd_refcnt++;  /* new linux thread  */
+   else
+   ishm_proctable->thrd_refcnt = 1;/* new linux process */
+
+   /*
+* if this ODP thread is actually a new linux process, (as opposed
+* to a pthread), i.e, we just forked, then all shmem blocks
+* of the parent process are mapped into this child by inheritance.
+* (The process local table is inherited as well). We hence have to
+* increase the process refcount for each of the inherited mappings:
+*/
+   if (syscall(SYS_gettid) == getpid()) {
+   for (i = 0; i < ishm_proctable->nb_entries; i++) {
+   block_index = ishm_proctable->entry[i].block_index;
+   ishm_tbl->block[block_index].refcnt++;
+   }
+   }
+
+   odp_spinlock_unlock(_tbl->lock);
+   return 0;
+}
+
 int 

[lng-odp] [API-NEXT PATCHv9 03/15] linux-gen: _ishm: create description file for external memory sharing

[Christophe Milard]

A new flag called _ODP_ISHM_EXPORT is added to _ishm. When this flag is
specified at reserve() time, an extra file
("/tmp/odp--shm-", where  is the process ID of the
main ODP instatiation process and  is the block name given at
reserve time) is created, describing to the underlying block attributes.
This file is meant to be used by processes external to ODP willing to
share this memory.

Signed-off-by: Christophe Milard 
---
 platform/linux-generic/_ishm.c  | 74 ++---
 platform/linux-generic/include/_ishm_internal.h |  1 +
 2 files changed, 68 insertions(+), 7 deletions(-)

diff --git a/platform/linux-generic/_ishm.c b/platform/linux-generic/_ishm.c
index 3371d94..e3ab8b4 100644
--- a/platform/linux-generic/_ishm.c
+++ b/platform/linux-generic/_ishm.c
@@ -99,6 +99,14 @@
 #define ISHM_FILENAME_NORMAL_PAGE_DIR "/tmp"
 
 /*
+ * when the memory is to be shared with an external entity (such as another
+ * ODP instance or an OS process not part of this ODP instance) then a
+ * export file is created describing the exported memory: this defines the
+ * location and the filename format of this description file
+ */
+#define ISHM_EXPTNAME_FORMAT "/tmp/odp-%d-shm-%s"
+
+/*
  * At worse case the virtual space gets so fragmented that there is
  * a unallocated fragment between each allocated fragment:
  * In that case, the number of fragments to take care of is twice the
@@ -107,6 +115,17 @@
 #define ISHM_NB_FRAGMNTS (ISHM_MAX_NB_BLOCKS * 2 + 1)
 
 /*
+ * when a memory block is to be exported outside its ODP instance,
+ * an block 'attribute file' is created in /tmp/odp--shm-.
+ * The information given in this file is according to the following:
+ */
+#define EXPORT_FILE_LINE1_FMT "ODP exported shm block info:"
+#define EXPORT_FILE_LINE2_FMT "ishm_blockname: %s"
+#define EXPORT_FILE_LINE3_FMT "file: %s"
+#define EXPORT_FILE_LINE4_FMT "length: %" PRIu64
+#define EXPORT_FILE_LINE5_FMT "flags: %" PRIu32
+#define EXPORT_FILE_LINE6_FMT "align: %" PRIu32
+/*
  * A fragment describes a piece of the shared virtual address space,
  * and is allocated only when allocation is done with the _ODP_ISHM_SINGLE_VA
  * flag:
@@ -136,6 +155,7 @@ typedef struct ishm_fragment {
 typedef struct ishm_block {
char name[ISHM_NAME_MAXLEN];/* name for the ishm block (if any) */
char filename[ISHM_FILENAME_MAXLEN]; /* name of the .../odp-* file  */
+   char exptname[ISHM_FILENAME_MAXLEN]; /* name of the export file */
int  main_odpthread; /* The thread which did the initial reserve*/
uint32_t user_flags; /* any flags the user want to remember.*/
uint32_t flags;  /* block creation flags.   */
@@ -380,7 +400,8 @@ static void free_fragment(ishm_fragment_t *fragmnt)
  * or /mnt/huge/odp-- (for huge pages)
  * Return the new file descriptor, or -1 on error.
  */
-static int create_file(int block_index, int huge, uint64_t len)
+static int create_file(int block_index, int huge, uint64_t len,
+  uint32_t flags, uint32_t align)
 {
char *name;
int  fd;
@@ -388,6 +409,7 @@ static int create_file(int block_index, int huge, uint64_t 
len)
char seq_string[ISHM_FILENAME_MAXLEN];   /* used to construct filename*/
char filename[ISHM_FILENAME_MAXLEN];/* filename in /tmp/ or /mnt/huge */
int  oflag = O_RDWR | O_CREAT | O_TRUNC; /* flags for open*/
+   FILE *export_file;
 
new_block = _tbl->block[block_index];
name = new_block->name;
@@ -429,9 +451,48 @@ static int create_file(int block_index, int huge, uint64_t 
len)
 
strncpy(new_block->filename, filename, ISHM_FILENAME_MAXLEN - 1);
 
+   /* if _ODP_ISHM_EXPORT is set, create a description file for
+* external ref:
+*/
+   if (flags & _ODP_ISHM_EXPORT) {
+   snprintf(new_block->exptname, ISHM_FILENAME_MAXLEN,
+ISHM_EXPTNAME_FORMAT,
+odp_global_data.main_pid,
+(name && name[0]) ? name : seq_string);
+   export_file = fopen(new_block->exptname, "w");
+   if (export_file == NULL) {
+   ODP_ERR("open failed: err=%s.\n",
+   strerror(errno));
+   new_block->exptname[0] = 0;
+   } else {
+   fprintf(export_file, EXPORT_FILE_LINE1_FMT "\n");
+   fprintf(export_file, EXPORT_FILE_LINE2_FMT "\n",  name);
+   fprintf(export_file, EXPORT_FILE_LINE3_FMT "\n",
+   new_block->filename);
+   fprintf(export_file, EXPORT_FILE_LINE4_FMT "\n", len);
+   fprintf(export_file, EXPORT_FILE_LINE5_FMT "\n", flags);
+   fprintf(export_file, EXPORT_FILE_LINE6_FMT "\n", align);
+
+   fclose(export_file);
+ 

[lng-odp] [API-NEXT PATCHv9 02/15] linux-gen: init: removing possible obsolete ODP files at startup

[Christophe Milard]

When an ODP program is killed, some odp files may remain in /tmp and
the huge page mount point. As signal KILL cannot be caught, there is not
much one can do to prevent that.
But when an new odp session is started, all files prefixed with the opd
prefix ("odp--") can be safely removed as the PID is unique and
therefore, there cannot be another ODP instance with the same PID.
This patch does this cleanup at startup.

Signed-off-by: Christophe Milard 
---
 platform/linux-generic/odp_init.c | 60 +++
 1 file changed, 60 insertions(+)

diff --git a/platform/linux-generic/odp_init.c 
b/platform/linux-generic/odp_init.c
index d4a8e09..6e58ab6 100644
--- a/platform/linux-generic/odp_init.c
+++ b/platform/linux-generic/odp_init.c
@@ -10,15 +10,71 @@
 #include 
 #include 
 #include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+
+#define _ODP_FILES_FMT "odp-%d-"
+#define _ODP_TMPDIR"/tmp"
 
 struct odp_global_data_s odp_global_data;
 
+/* remove all files staring with "odp-" from a directory "dir" */
+static int cleanup_files(const char *dirpath, int odp_pid)
+{
+   struct dirent *e;
+   DIR *dir;
+   char prefix[PATH_MAX];
+   char *fullpath;
+   int d_len = strlen(dirpath);
+   int p_len;
+   int f_len;
+
+   dir = opendir(dirpath);
+   if (!dir) {
+   /* ok if the dir does not exist. no much to delete then! */
+   ODP_DBG("opendir failed for %s: %s\n",
+   dirpath, strerror(errno));
+   return 0;
+   }
+   snprintf(prefix, PATH_MAX, _ODP_FILES_FMT, odp_pid);
+   p_len = strlen(prefix);
+   while ((e = readdir(dir)) != NULL) {
+   if (strncmp(e->d_name, prefix, p_len) == 0) {
+   f_len = strlen(e->d_name);
+   fullpath = malloc(d_len + f_len + 2);
+   if (fullpath == NULL) {
+   closedir(dir);
+   return -1;
+   }
+   snprintf(fullpath, PATH_MAX, "%s/%s",
+dirpath, e->d_name);
+   ODP_DBG("deleting obsolete file: %s\n", fullpath);
+   if (unlink(fullpath))
+   ODP_ERR("unlink failed for %s: %s\n",
+   fullpath, strerror(errno));
+   free(fullpath);
+   }
+   }
+   closedir(dir);
+
+   return 0;
+}
+
 int odp_init_global(odp_instance_t *instance,
const odp_init_t *params,
const odp_platform_init_t *platform_params)
 {
+   char *hpdir;
+
memset(_global_data, 0, sizeof(struct odp_global_data_s));
odp_global_data.main_pid = getpid();
+   cleanup_files(_ODP_TMPDIR, odp_global_data.main_pid);
+
if (platform_params)
odp_global_data.ipc_ns = platform_params->ipc_ns;
 
@@ -49,6 +105,10 @@ int odp_init_global(odp_instance_t *instance,
ODP_ERR("ODP system_info init failed.\n");
goto init_failed;
}
+   hpdir = odp_global_data.hugepage_info.default_huge_page_dir;
+   /* cleanup obsolete huge page files, if any */
+   if (hpdir)
+   cleanup_files(hpdir, odp_global_data.main_pid);
stage = SYSINFO_INIT;
 
if (_odp_fdserver_init_global()) {
-- 
2.7.4

[lng-odp] [API-NEXT PATCHv9 00/15] using _ishm as north API mem allocator

[Christophe Milard]

since v8:
- commit msg typo fix (Maxim)
- use snprintf() instead of sprintf() (Maxim)
- better handling of error cases (Maxim)

since v7:
- Fix not to use single-va memory when the requirement alignment exactely
matches the page size (Maxim)
- patch to clean-up all odp--* files in /tmp and in the huge
page mount point at startup (in case some previous killed / crashed
ODP instance did not clean properly.) (Maxim)

since v6:
- All points according to Petri's request i.e.:
Odp_shm_find_external() changed again: now odp_shm_import().
Function description updated.

since v5:
-fixed doxygen issue (Bill)

since v4:
- All points of v3 according to Petri's request i.e.:
-All API patches merges as one single patch.
-lock flag removed: memory is now always locked.
-alignement and flag inherited from remote ODP instance when sharing
 memory.
-function name changed to: odp_shm_find_external().

since v3:
-Comments from Petri addressed, with exceptions for:
-Patches related to the API not merged as they are part of different
 functions.
-Lock flag not removed: Why wouldn't any ODP application being allowed
 to reserve memory for slower (control or other things) businnes.
 Should really ALL shared memory bee locked?
-Alignment and flag parameter of odp_shm_reserve_external() kept:
 The alignment (and some flags) affect virtual memory, which is different
 between for each ODP instance.
-function name odp_shm_reserve_external() kept, as this function does
 return a new handle (which requires free() as any handle would)
These 4 items are sheduled for arch call today

since v2:
-some minor changes on the doc (Bill)
-doc for the ODP_SHM_EXPORT flag (Christophe)
-reduction of memory reguirement for shm tests (Mattias)

since v1:
-flag _ODP_SHM_PROC_NOCREAT and _ODP_SHM_O_EXCL get new values
(but remain useless: Should be removed when IPC is updated)  (Maxim)

-In get_ishm_flags(), odp_shm_capability() local variable flgs renamed
(for be better distinction from other "flags" variable. (Maxim)

-Added doc updates with shm api extensions.

This Patch series aims at using _ishm as north API memory allocator.
odp_shared_memory.c just becomes a wrapper around _ishm.
_ishm supports "process mode", i.e. memory allocated  with _ishm
is sharable by all ODP threads of a given ODP instance, regardless of
thread type (e.g. process) or thread creation time (for time).

NOTE: This patch series will break IPC: This is due to the fact that
IPC relied on a "special case" in the former memory allocator that broke
ODP instance scoping. I don't think this should be kept.
I have included in this patch series a function to share memory between
designated ODP instances. If we want to have IPC, it should use that.

Christophe Milard (15):
  linux-gen: _ishm: fix for alignment request matching page size
  linux-gen: init: removing possible obsolete ODP files at startup
  linux-gen: _ishm: create description file for external memory sharing
  linux-gen: _ishm: allow memory alloc/free at global init/term
  linux-gen: use ishm as north API mem allocator
  linux-gen: push internal flag definition
  api: shm: add flags to shm_reserve and function to find external mem
  linux-gen: shm: new ODP_SHM_SINGLE_VA flag implementation
  test: api: shmem: new proper tests for shm API
  linux-gen: _ishm: adding function to map memory from other ODP
  linux-gen: shm: add flag and function to share memory between ODP
instances
  test: linux-gen: api: shmem: test sharing memory between ODP instances
  linux-gen: _ishm: cleaning remaining block at odp_term_global
  linux_gen: _ishm: decreasing the number of error messages when no huge
pages
  doc: updating docs for the shm interface extension

 doc/users-guide/users-guide.adoc   |  68 +-
 include/odp/api/spec/shared_memory.h   |  46 +-
 platform/linux-generic/_ishm.c | 460 ++
 platform/linux-generic/include/_ishm_internal.h|   6 +
 platform/linux-generic/include/odp_internal.h  |   5 -
 platform/linux-generic/include/odp_shm_internal.h  |   4 +-
 platform/linux-generic/odp_init.c  |  79 ++-
 platform/linux-generic/odp_shared_memory.c | 412 ++--
 test/common_plat/validation/api/shmem/shmem.c  | 687 -
 test/common_plat/validation/api/shmem/shmem.h  |   5 +-
 test/linux-generic/validation/api/shmem/.gitignore |   3 +-
 .../linux-generic/validation/api/shmem/Makefile.am |  22 +-
 .../validation/api/shmem/shmem_linux.c | 220 +--
 .../api/shmem/{shmem_odp.c => shmem_odp1.c}|  10 +-
 .../api/shmem/{shmem_odp.h => shmem_odp1.h}|   0
 .../validation/api/shmem/shmem_odp2.c  |  95 +++
 .../validation/api/shmem/shmem_odp2.h  |   7 +
 17 files changed, 1526 insertions(+), 603 

[lng-odp] [API-NEXT PATCHv9 01/15] linux-gen: _ishm: fix for alignment request matching page size

[Christophe Milard]

There is no reason to toggle the _ODP_ISHM_SINGLE_VA flag
when the alignment exactly matches the page size.
This just results in wasting the odp-common virtual space

Signed-off-by: Christophe Milard 
---
 platform/linux-generic/_ishm.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/platform/linux-generic/_ishm.c b/platform/linux-generic/_ishm.c
index f4aa6d3..3371d94 100644
--- a/platform/linux-generic/_ishm.c
+++ b/platform/linux-generic/_ishm.c
@@ -756,7 +756,7 @@ int _odp_ishm_reserve(const char *name, uint64_t size, int 
fd,
 * the same address every where, otherwise alignment may be
 * be wrong for some process */
hp_align = align;
-   if (hp_align < odp_sys_huge_page_size())
+   if (hp_align <= odp_sys_huge_page_size())
hp_align = odp_sys_huge_page_size();
else
flags |= _ODP_ISHM_SINGLE_VA;
@@ -777,7 +777,7 @@ int _odp_ishm_reserve(const char *name, uint64_t size, int 
fd,
 * size then we have to make sure the block will be mapped at
 * the same address every where, otherwise alignment may be
 * be wrong for some process */
-   if (align < odp_sys_page_size())
+   if (align <= odp_sys_page_size())
align = odp_sys_page_size();
else
flags |= _ODP_ISHM_SINGLE_VA;
-- 
2.7.4

Re: [lng-odp] [API-NEXT PATCHv8 02/15] linux-gen: init: removing possible obsolete ODP files at startup

[Christophe Milard]

On 24 November 2016 at 15:36, Maxim Uvarov  wrote:
> On 11/24/16 11:24, Christophe Milard wrote:
>>
>> When an ODP program is killed, some odp files may remain in /tmp and
>> the huge page mount point. As signal KILL cannot be caught, there is not
>> much one can do to prevent that.
>> But when an new odp session is started, all files prefixed with the opd
>> prefix ("odp--") can be safely removed as the PID is unique and
>> therefore, there cannot be another ODP instance with the same PID.
>> This patch does this cleanup at startup.
>>
>> Signed-off-by: Christophe Milard 
>> ---
>>   platform/linux-generic/odp_init.c | 51
>> +++
>>   1 file changed, 51 insertions(+)
>>
>> diff --git a/platform/linux-generic/odp_init.c
>> b/platform/linux-generic/odp_init.c
>> index d4a8e09..6f63a4a 100644
>> --- a/platform/linux-generic/odp_init.c
>> +++ b/platform/linux-generic/odp_init.c
>> @@ -10,15 +10,62 @@
>>   #include 
>>   #include 
>>   #include 
>> +#include 
>> +#include 
>> +#include 
>> +#include 
>> +#include 
>> +#include 
>> +#include 
>> +
>> +#define _ODP_FILES_FMT "odp-%d-"
>> +#define _ODP_TMPDIR"/tmp"
>> struct odp_global_data_s odp_global_data;
>>   +/* remove all files staring with "odp-" from a directory "dir" */
>> +static int cleanup_files(const char *dirpath, int odp_pid)
>> +{
>> +   struct dirent *e;
>> +   DIR *dir;
>> +   char prefix[PATH_MAX];
>> +   char *fullpath;
>> +   int d_len = strlen(dirpath);
>> +   int p_len;
>> +   int f_len;
>> +
>> +   dir = opendir(dirpath);
>> +   sprintf(prefix, _ODP_FILES_FMT, odp_pid);
>> +   p_len = strlen(prefix);
>> +   while ((e = readdir(dir)) != NULL) {
>> +   if (strncmp(e->d_name, prefix, p_len) == 0) {
>> +   f_len = strlen(e->d_name);
>> +   fullpath = malloc(d_len + f_len + 2);
>
>
>> +   if (fullpath == NULL)
>
> close(dir)

Argh!! =>V9

>
>> +   return -1;
>> +   sprintf(fullpath, "%s/%s", dirpath, e->d_name);
>
>
> snprintf

what for? fullpath is malloc'ed accordinaly and will neveer exceed
2*PATH_MAX... What n value would you give to snprintf()?

Christophe.
>
>
>> +   ODP_DBG("deleting obsolete file: %s\n", fullpath);
>> +   if (unlink(fullpath))
>> +   ODP_ERR("unlink failed for %s: %s\n",
>> +   fullpath, strerror(errno));
>> +   free(fullpath);
>> +   }
>> +   }
>> +   closedir(dir);
>> +
>> +   return 0;
>> +}
>> +
>>   int odp_init_global(odp_instance_t *instance,
>> const odp_init_t *params,
>> const odp_platform_init_t *platform_params)
>>   {
>> +   char *hpdir;
>> +
>> memset(_global_data, 0, sizeof(struct odp_global_data_s));
>> odp_global_data.main_pid = getpid();
>> +   cleanup_files(_ODP_TMPDIR, odp_global_data.main_pid);
>> +
>> if (platform_params)
>> odp_global_data.ipc_ns = platform_params->ipc_ns;
>>   @@ -49,6 +96,10 @@ int odp_init_global(odp_instance_t *instance,
>> ODP_ERR("ODP system_info init failed.\n");
>> goto init_failed;
>> }
>> +   hpdir = odp_global_data.hugepage_info.default_huge_page_dir;
>> +   /* cleanup obsolete huge page files, if any */
>> +   if (hpdir)
>> +   cleanup_files(hpdir, odp_global_data.main_pid);
>> stage = SYSINFO_INIT;
>> if (_odp_fdserver_init_global()) {
>
>

Re: [lng-odp] [API-NEXT PATCHv8 02/15] linux-gen: init: removing possible obsolete ODP files at startup

[Maxim Uvarov]

On 11/24/16 11:24, Christophe Milard wrote:

When an ODP program is killed, some odp files may remain in /tmp and
the huge page mount point. As signal KILL cannot be caught, there is not
much one can do to prevent that.
But when an new odp session is started, all files prefixed with the opd
prefix ("odp--") can be safely removed as the PID is unique and
therefore, there cannot be another ODP instance with the same PID.
This patch does this cleanup at startup.

Signed-off-by: Christophe Milard 
---
  platform/linux-generic/odp_init.c | 51 +++
  1 file changed, 51 insertions(+)

diff --git a/platform/linux-generic/odp_init.c 
b/platform/linux-generic/odp_init.c
index d4a8e09..6f63a4a 100644
--- a/platform/linux-generic/odp_init.c
+++ b/platform/linux-generic/odp_init.c
@@ -10,15 +10,62 @@
  #include 
  #include 
  #include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+
+#define _ODP_FILES_FMT "odp-%d-"
+#define _ODP_TMPDIR"/tmp"
  
  struct odp_global_data_s odp_global_data;
  
+/* remove all files staring with "odp-" from a directory "dir" */

+static int cleanup_files(const char *dirpath, int odp_pid)
+{
+   struct dirent *e;
+   DIR *dir;
+   char prefix[PATH_MAX];
+   char *fullpath;
+   int d_len = strlen(dirpath);
+   int p_len;
+   int f_len;
+
+   dir = opendir(dirpath);
+   sprintf(prefix, _ODP_FILES_FMT, odp_pid);
+   p_len = strlen(prefix);
+   while ((e = readdir(dir)) != NULL) {
+   if (strncmp(e->d_name, prefix, p_len) == 0) {
+   f_len = strlen(e->d_name);
+   fullpath = malloc(d_len + f_len + 2);



+   if (fullpath == NULL)

close(dir)


+   return -1;
+   sprintf(fullpath, "%s/%s", dirpath, e->d_name);


snprintf


+   ODP_DBG("deleting obsolete file: %s\n", fullpath);
+   if (unlink(fullpath))
+   ODP_ERR("unlink failed for %s: %s\n",
+   fullpath, strerror(errno));
+   free(fullpath);
+   }
+   }
+   closedir(dir);
+
+   return 0;
+}
+
  int odp_init_global(odp_instance_t *instance,
const odp_init_t *params,
const odp_platform_init_t *platform_params)
  {
+   char *hpdir;
+
memset(_global_data, 0, sizeof(struct odp_global_data_s));
odp_global_data.main_pid = getpid();
+   cleanup_files(_ODP_TMPDIR, odp_global_data.main_pid);
+
if (platform_params)
odp_global_data.ipc_ns = platform_params->ipc_ns;
  
@@ -49,6 +96,10 @@ int odp_init_global(odp_instance_t *instance,

ODP_ERR("ODP system_info init failed.\n");
goto init_failed;
}
+   hpdir = odp_global_data.hugepage_info.default_huge_page_dir;
+   /* cleanup obsolete huge page files, if any */
+   if (hpdir)
+   cleanup_files(hpdir, odp_global_data.main_pid);
stage = SYSINFO_INIT;
  
  	if (_odp_fdserver_init_global()) {

Re: [lng-odp] [API-NEXT PATCHv8 01/15] linux-gen: _ishm: fix for alignment request matching page size

[Maxim Uvarov]

On 11/24/16 11:24, Christophe Milard wrote:

There is no reason to toggle the _ODP_ISHM_SINGLE_VA flag
when the alignment exactely


exactly



  matches the page size.
This just results in wasting the odp-common virtual space

Signed-off-by: Christophe Milard 
---
  platform/linux-generic/_ishm.c | 4 ++--
  1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/platform/linux-generic/_ishm.c b/platform/linux-generic/_ishm.c
index f4aa6d3..3371d94 100644
--- a/platform/linux-generic/_ishm.c
+++ b/platform/linux-generic/_ishm.c
@@ -756,7 +756,7 @@ int _odp_ishm_reserve(const char *name, uint64_t size, int 
fd,
 * the same address every where, otherwise alignment may be
 * be wrong for some process */
hp_align = align;
-   if (hp_align < odp_sys_huge_page_size())
+   if (hp_align <= odp_sys_huge_page_size())
hp_align = odp_sys_huge_page_size();
else
flags |= _ODP_ISHM_SINGLE_VA;
@@ -777,7 +777,7 @@ int _odp_ishm_reserve(const char *name, uint64_t size, int 
fd,
 * size then we have to make sure the block will be mapped at
 * the same address every where, otherwise alignment may be
 * be wrong for some process */
-   if (align < odp_sys_page_size())
+   if (align <= odp_sys_page_size())
align = odp_sys_page_size();
else
flags |= _ODP_ISHM_SINGLE_VA;

Re: [lng-odp] [API-NEXT PATCH] api: ipsec: added IPSEC API

[Savolainen, Petri (Nokia - FI/Espoo)]

> -Original Message-
> From: Bala Manoharan [mailto:bala.manoha...@linaro.org]
> Sent: Thursday, November 24, 2016 9:50 AM
> To: Savolainen, Petri (Nokia - FI/Espoo)  labs.com>
> Cc: lng-odp-forward 
> Subject: Re: [lng-odp] [API-NEXT PATCH] api: ipsec: added IPSEC API
> 
> Regards,
> Bala
> 
> 
> On 23 November 2016 at 16:09, Petri Savolainen
>  wrote:
> > Added definitions for a look-a-side IPSEC offload API. In addition to
> > IPSEC packet transformations, it also supports:
> > * inbound SA look up
> > * outbound IP fragmentation
> >
> > Signed-off-by: Petri Savolainen 
> > ---
> >
> > Changes to draft:
> > * renamed odp_ipsec_proto_t to renamed odp_ipsec_protocol_t
> > * specify that lifetime sec limit is from the SA creation
> > * added odp_ipsec_sa_context()
> > * pool for output packets is the same as packet input pool
> > * added antireplay check and protocol error codes
> > * specified which input / output packet offsets and flags are set
> > * moved sync/async mode selection to global config (odp_ipsec_config())
> > * added IPSEC capability to aid mode selection
> > * specify that also packet user area is copied from input to output
> packet
> >
> >
> >  include/odp/api/spec/event.h   |   2 +-
> >  include/odp/api/spec/ipsec.h   | 856
> +
> >  include/odp_api.h  |   1 +
> >  platform/Makefile.inc  |   1 +
> >  platform/linux-generic/Makefile.am |   2 +
> >  platform/linux-generic/include/odp/api/ipsec.h |  36 +
> >  .../include/odp/api/plat/event_types.h |   1 +
> >  .../include/odp/api/plat/ipsec_types.h |  39 +
> >  8 files changed, 937 insertions(+), 1 deletion(-)
> >  create mode 100644 include/odp/api/spec/ipsec.h
> >  create mode 100644 platform/linux-generic/include/odp/api/ipsec.h
> >  create mode 100644 platform/linux-
> generic/include/odp/api/plat/ipsec_types.h
> >

> > +
> > +/**
> > + * IPSEC capability
> > + */
> > +typedef struct odp_ipsec_capability_t {
> > +   /** Maximum number of IPSEC SAs */
> > +   uint32_t max_num_sa;
> > +
> > +   /** Synchronous IPSEC operation mode (ODP_IPSEC_OP_MODE_SYNC)
> support
> > +*
> > +*  0: Synchronous mode is not supported
> > +*  1: Synchronous mode is supported
> > +*  2: Synchronous mode is supported and preferred
> > +*/
> > +   int op_mode_sync;
> > +
> > +   /** Asynchronous IPSEC operation mode (ODP_IPSEC_OP_MODE_ASYNC)
> support
> > +*
> > +*  0: Asynchronous mode is not supported
> > +*  1: Asynchronous mode is supported
> > +*  2: Asynchronous mode is supported and preferred
> > +*/
> > +   int op_mode_async;
> 
> Not sure if this is something which should be dictated by the application.
> It might be possible in some cases depending on the packet size sync
> processing might be faster than async.
> Should we leave this decision to the implementation similar to what is
> being done in crypto operations?

This is capability. An implementation may not e.g. support sync mode at all, or 
prefer async mode.

I think that crypto API style is problematic, since implementation may decide 
packet by packet to return output packet right away or through queues. 
Application must have code for both sync and async paths, as well as 
potentially reordered packets that are received through different paths.

In this API, an async mode implementation may process some packets on the CPU 
and some on external HW, as long as it can ensure correct packet order in the 
destination queue. Packet ordering burden is on implementation side and 
application is simplified.

Later on, we could consider sync/async selection for in- and outbound 
directions independently (now both dir are either sync or async).

-Petri

Re: [lng-odp] [API-NEXT PATCH] api: ipsec: added IPSEC API

[Savolainen, Petri (Nokia - FI/Espoo)]

Hi,

You missed to reply to the list.

No HTML please. Readability of a conversation is especially bad when you reply 
in HTML to patches... 


From: Honnappa Nagarahalli [mailto:honnappa.nagaraha...@linaro.org] 
Sent: Wednesday, November 23, 2016 11:34 PM
To: Savolainen, Petri (Nokia - FI/Espoo) 
Subject: Re: [lng-odp] [API-NEXT PATCH] api: ipsec: added IPSEC API



On 23 November 2016 at 04:39, Petri Savolainen  
wrote:
Added definitions for a look-a-side IPSEC offload API. In addition to
IPSEC packet transformations, it also supports:
* inbound SA look up
* outbound IP fragmentation

Signed-off-by: Petri Savolainen 
---

Changes to draft:
* renamed odp_ipsec_proto_t to renamed odp_ipsec_protocol_t
* specify that lifetime sec limit is from the SA creation
* added odp_ipsec_sa_context()
* pool for output packets is the same as packet input pool
* added antireplay check and protocol error codes
* specified which input / output packet offsets and flags are set
* moved sync/async mode selection to global config (odp_ipsec_config())
* added IPSEC capability to aid mode selection
* specify that also packet user area is copied from input to output packet


 include/odp/api/spec/event.h                       |   2 +-
 include/odp/api/spec/ipsec.h                       | 856 +
 include/odp_api.h                                  |   1 +
 platform/Makefile.inc                              |   1 +
 platform/linux-generic/Makefile.am                 |   2 +
 platform/linux-generic/include/odp/api/ipsec.h     |  36 +
 .../include/odp/api/plat/event_types.h             |   1 +
 .../include/odp/api/plat/ipsec_types.h             |  39 +
 8 files changed, 937 insertions(+), 1 deletion(-)
 create mode 100644 include/odp/api/spec/ipsec.h
 create mode 100644 platform/linux-generic/include/odp/api/ipsec.h
 create mode 100644 platform/linux-generic/include/odp/api/plat/ipsec_types.h
+
+/**
+ * Query IPSEC capabilities
+ *
+ * Outputs IPSEC capabilities on success.
+ *
+ * @param[out] capa   Pointer to capability structure for output
+ *
+ * @retval 0 on success
+ * @retval <0 on failure
+ */
+int odp_ipsec_capability(odp_ipsec_capability_t *capa);

Return value type needs to be int32_t
 
---

No, we used int as the common success/fail return code. It's also used for (a 
small )number of objects 
(see e.g. odp_queue_deq_multi()). Int32_t or uint32_t is used when we return a 
number of bytes, or 
Something else that may be large (> thousands).

---


+
+/**
+ * Initialize IPSEC configuration options
+ *
+ * Initialize an odp_ipsec_config_t to its default values.
+ *
+ * @param[out] config  Pointer to IPSEC configuration structure
+ */
+void odp_ipsec_config_init(odp_ipsec_config_t *config);
+
+/**
+ * Global IPSEC configuration
+ *
+ * Initialize and configure IPSEC offload with global configuration options.
+ * This must be called before any SAs are created. Use odp_ipsec_capability()
+ * to examine which features and modes are supported.
+ *
+ * @param config   Pointer to IPSEC configuration structure
+ *
+ * @retval 0 on success
+ * @retval <0 on failure
+ *
+ * @see odp_ipsec_capability(), odp_ipsec_config_init()
+ */
+int odp_ipsec_config(const odp_ipsec_config_t *config);
 
Return value type needs to be int32_t 

---
Same here.
---


+/**
+ * Destroy IPSEC SA
+ *
+ * Destroy an unused IPSEC SA. Result is undefined if the SA is being used
+ * (i.e. asynchronous operation is in progress).
+ *
+ * @param sa      IPSEC SA to be destroyed
+ *
+ * @retval 0      On success
+ * @retval <0     On failure
+ *
+ * @see odp_ipsec_sa_create()
+ */
+int odp_ipsec_sa_destroy(odp_ipsec_sa_t sa);

Return value type needs to be int32_t

---
Same here.
---
 
+
+/**
+ * Printable format of odp_ipsec_sa_t
+ *
+ * @param sa      IPSEC SA handle
+ *
+ * @return uint64_t value that can be used to print/display this handle
+ */
+uint64_t odp_ipsec_sa_to_u64(odp_ipsec_sa_t sa);
+
+/**
+ * IPSEC operation level options
+ *
+ * These may be used to override some SA level options
+ */
+typedef struct odp_ipsec_op_opt_t {
+       /** Fragmentation mode */
+       odp_ipsec_frag_mode_t mode;
+
+} odp_ipsec_op_opt_t;
+
+/** IPSEC operation status has no errors */
+#define ODP_IPSEC_OK 0
+
+/** IPSEC operation status */
+typedef union odp_ipsec_status_t {
+       /** Error flags */
+       struct {
+               /** Protocol error. Not a valid ESP or AH packet. */
+               uint32_t 

Re: [lng-odp] [API-NEXT PATCHv4 1/7] api: random: replace use_entropy with odp_rand_kind parameter

[Savolainen, Petri (Nokia - FI/Espoo)]

From: Bill Fischofer [mailto:bill.fischo...@linaro.org] 
Sent: Wednesday, November 23, 2016 4:05 PM
To: Savolainen, Petri (Nokia - FI/Espoo) 
Cc: lng-odp@lists.linaro.org
Subject: Re: [lng-odp] [API-NEXT PATCHv4 1/7] api: random: replace use_entropy 
with odp_rand_kind parameter



On Wed, Nov 23, 2016 at 8:02 AM, Savolainen, Petri (Nokia - FI/Espoo) 
 wrote:


> -Original Message-
> From: lng-odp [mailto:lng-odp-boun...@lists.linaro.org] On Behalf Of Bill
> Fischofer
> Sent: Thursday, November 03, 2016 2:42 AM
> To: lng-odp@lists.linaro.org
> Subject: [lng-odp] [API-NEXT PATCHv4 1/7] api: random: replace use_entropy
> with odp_rand_kind parameter
>
> Address bug https://bugs.linaro.org/show_bug.cgi?id=2557 by replacing the
> use_entropy parameter with a more comprehensive enum that specifies the
> kind of random data requested.
>
> Signed-off-by: Bill Fischofer 
> ---
> Changes in v4:
> - Normalize random API signatures with other ODP APIs
> - Add new odp_random_seeded_data() API for repeatable random data
> generation
> - Add additional tests for new odp_random_seeded_data() API
> - Break out crypto section of User Guide to its own sub-document
> - Add User Guide docuemntation for ODP random data API.
>
> Changes in v3:
> - Address commments by Petri
> - Rename ODP_RAND_NORMAL to ODP_RANDOM_BASIC to avoid confusion with the
> mathematical term "normal"
>
>  include/odp/api/spec/random.h | 31 ++-
>  1 file changed, 26 insertions(+), 5 deletions(-)
>
> diff --git a/include/odp/api/spec/random.h b/include/odp/api/spec/random.h
> index 00fa15b..62f2376 100644
> --- a/include/odp/api/spec/random.h
> +++ b/include/odp/api/spec/random.h
> @@ -24,18 +24,39 @@ extern "C" {
>   */
>
>  /**
> + * Random kind selector
> + */
> +typedef enum {
> +     /** Basic random, presumably pseudo-random generated by SW */
> +     ODP_RANDOM_BASIC,
> +     /** Cryptographic quality random */
> +     ODP_RANDOM_CRYPTO,
> +     /** True random, generated from a HW entropy source */
> +     ODP_RANDOM_TRUE,
> +} odp_random_kind_t;
> +
> +/**
>   * Generate random byte data
>   *
> - * @param[out]    buf   Output buffer
> - * @param         size  Size of output buffer
> - * @param use_entropy   Use entropy
> + * The intent in supporting different kinds of random data is to allow
> + * tradeoffs between performance and the quality of random data needed.
> The
> + * assumption is that basic random is cheap while true random is
> relatively
> + * expensive in terms of time to generate, with cryptographic random
> being
> + * something in between. Implementations that support highly efficient
> true
> + * random are free to use this for all requested kinds. So it is always
> + * permissible to "upgrade" a random data request, but never to
> "downgrade"
> + * such requests.
>   *
> - * @todo Define the implication of the use_entropy parameter
> + * @param[out]    buf   Output buffer
> + * @param         len   Length of output buffer in bytes
> + * @param         kind  Specifies the type of random data required.
> Request
> + *                      is expected to fail if the implementation is
> unable to
> + *                      provide the requested type.
>   *
>   * @return Number of bytes written
>   * @retval <0 on failure
>   */
> -int32_t odp_random_data(uint8_t *buf, int32_t size, odp_bool_t
> use_entropy);
> +int odp_random_data(uint8_t *buf, uint32_t len, odp_random_kind_t kind);
Cannot change int types. Len and return value need to be compared commonly, and 
signed vs. unsigned comparison produces a compiler warning. Also C spec specify 
that 'int' is at least 16 bits, which mean that it may the that on some 
systems. Number of bytes are stored into 32 bit variables throughout the ODP 
API.

ODP does not support 16-bit systems. So on any system ODP would actually be 
implemented on int will be 32-bits. We use int as a general return type for 
RCs, which is why I made this change. Changing len to uint32_t is consistent 
with other ODP APIs since you cannot request a negative amount of data (which 
would be possible with signed input).  The comparison works fine on both GCC 
and clang. What error do you see?

... HTML comes through still ...

No, ODP API does not dictate which CPU architecture must be used. It may run on 
a 16 bit system with 16 bit addresses and 32 bit integers, or on a 32 bit 
system with 32 bit addresses and 16 bit integers, or ...

We do API definition against C spec, which contains this (as an example of int 
size):
minimum value for an object of type int
INT_MIN -32767 // -(2^15 - 1)
— maximum value for an object of type int
INT_MAX +32767 // 2^15 - 1


int random_data(uint8_t *buf, uint32_t len)
{
(void)buf;

return len;
}

int main(int argc, char *argv[])
{
uint32_t len = 32;
uint8_t buf[len];

if (random_data(buf, len) != 

Re: [lng-odp] lng-odp at a text only list.

[Savolainen, Petri (Nokia - FI/Espoo)]

Could bugzilla spam a dedicated lng-odp-bugs / lng-odp-reports / etc list ?

We'd need only one strictly plain text list - the one that humans use for 
patches.

-Petri


> -Original Message-
> From: lng-odp [mailto:lng-odp-boun...@lists.linaro.org] On Behalf Of Mike
> Holmes
> Sent: Tuesday, November 22, 2016 8:37 PM
> To: lng-odp 
> Subject: [lng-odp] lng-odp at a text only list.
> 
> All there is a problem with having this as a text only list, bugzilla
> posts updates in html format, I will see if that can be changed but we
> might need to revert this experiment.
> 
> Mike
> 
> --
> Mike Holmes
> Program Manager - Linaro Networking Group
> Linaro.org │ Open source software for ARM SoCs
> "Work should be fun and collaborative, the rest follows"

[lng-odp] [PATCH] test: validation: updating crypto auth capability check

[Balakrishna Garapati]

Add missing capability check on crypto auth algos

Signed-off-by: Balakrishna Garapati 
---
 test/common_plat/validation/api/crypto/odp_crypto_test_inp.c | 12 
 1 file changed, 12 insertions(+)

diff --git a/test/common_plat/validation/api/crypto/odp_crypto_test_inp.c 
b/test/common_plat/validation/api/crypto/odp_crypto_test_inp.c
index 4ac4a07..292fbd4 100644
--- a/test/common_plat/validation/api/crypto/odp_crypto_test_inp.c
+++ b/test/common_plat/validation/api/crypto/odp_crypto_test_inp.c
@@ -92,6 +92,12 @@ static void alg_test(odp_crypto_op_t op,
if (auth_alg == ODP_AUTH_ALG_NULL &&
!(capability.hw_auths.bit.null))
rc = -1;
+   if (auth_alg == ODP_AUTH_ALG_MD5_96 &&
+   !(capability.hw_auths.bit.null))
+   rc = -1;
+   if (auth_alg == ODP_AUTH_ALG_SHA256_128 &&
+   !(capability.hw_auths.bit.null))
+   rc = -1;
} else {
if (auth_alg == ODP_AUTH_ALG_AES128_GCM &&
!(capability.auths.bit.aes128_gcm))
@@ -99,6 +105,12 @@ static void alg_test(odp_crypto_op_t op,
if (auth_alg == ODP_AUTH_ALG_NULL &&
!(capability.auths.bit.null))
rc = -1;
+   if (auth_alg == ODP_AUTH_ALG_MD5_96 &&
+   !(capability.auths.bit.null))
+   rc = -1;
+   if (auth_alg == ODP_AUTH_ALG_SHA256_128 &&
+   !(capability.auths.bit.null))
+   rc = -1;
}
 
CU_ASSERT(!rc);
-- 
1.9.1