Re: [lng-odp] issues with usage of mellanox 100G NICs with ODP & ODP-DPDK

[Bill Fischofer]

Thanks. I've opened Bug https://bugs.linaro.org/show_bug.cgi?id=3657
to track this.

On Tue, Mar 6, 2018 at 1:36 PM, gyanesh patra  wrote:
> Hi,
> I am coming back with the same issue here. As the odp-dpdk is updated to
> the recent DPDK version inline with the ODP code base, now both ODP repo
> and odp-dpdk repos stops working with mellanox interfaces. Please let me
> know if this is a known issue. I tried to run with master branch,
> caterpillar branch with and without 'abi-compat'. The ODP and odp-dpdk code
> compilation process is kept equal. The dpdk is compiled with mlx flags and
> working properly.
>
> Here is the configuratin details and the errors i get while trying to run
> ./test/performance/odp_l2fwd use case with odp github code base (master
> branch and has the v1.18.0.0 tag).
>
> <>
> ubuntu@ubuntu:~# uname -a
> Linux ubuntu 4.4.0-112-generic #135-Ubuntu SMP Fri Jan 19 11:48:36 UTC 2018
> x86_64 x86_64 x86_64 GNU/Linux
>
> ubuntu@ubuntu:/home/gyanesh/dpdk-17.11# ./usertools/dpdk-devbind.py -s
>
> Network devices using DPDK-compatible driver
> 
> 
>
> Network devices using kernel driver
> ===
> :81:00.0 'MT27700 Family [ConnectX-4] 1013' if=enp129s0f0 drv=mlx5_core
> unused=
> :81:00.1 'MT27700 Family [ConnectX-4] 1013' if=enp129s0f1 drv=mlx5_core
> unused=
>
> <>
> ubuntu@ubuntu:/home/gyanesh/odp# git branch -a
>   caterpillar
> * master
>   remotes/origin/HEAD -> origin/master
>   remotes/origin/api-next
>   remotes/origin/caterpillar
>
> ubuntu@ubuntu:/home/gyanesh/odp# ./bootstrap
> ubuntu@ubuntu:/home/gyanesh/odp# ./configure
> --prefix=/home/gyanesh/odp/build
> --with-dpdk-path=/home/gyanesh/dpdk-17.11/x86_64-native-linuxapp-gcc/
> --enable-dpdk-zero-copy --disable-abi-compat --enable-debug-print
> --enable-debug --enable-helper-linux --disable-doxygen-doc
> --enable-helper-debug-print --enable-test-example --enable-test-helper
> --enable-test-perf --enable-test-vald LDFLAGS="-libverbs -lmlx5"
>
> ubuntu@ubuntu:/home/gyanesh/odp# make
> ubuntu@ubuntu:/home/gyanesh/odp# cd test/performance
>
> << performance testcase >>
> ubuntu@ubuntu:/home/gyanesh/odp/test/performance# export
> ODP_PLATFORM_PARAMS="-n 4 -m 15240"
> ubuntu@ubuntu:/home/gyanesh/odp/test/performance# ./odp_l2fwd -i 0,1 -c 2
> HW time counter freq: 240786 hz
>
> odp_system_info.c:100:default_huge_page_size():defaut hp size is 1048576 kB
> odp_system_info.c:100:default_huge_page_size():defaut hp size is 1048576 kB
> _ishm.c:1468:_odp_ishm_init_global():ishm: using dir /dev/shm
> _ishm.c:1484:_odp_ishm_init_global():Huge pages mount point is:
> /dev/hugepages
> _ishmphy.c:65:_odp_ishmphy_book_va():VA Reserved: 0x7f07dd70b000,
> len=0x6000
> _fdserver.c:284:_odp_fdserver_register_fd():FD client register: pid=43825
> key=0, fd=3
> _fdserver.c:468:handle_request():storing {ctx=1, key=0}->fd=5
> _fdserver.c:284:_odp_fdserver_register_fd():FD client register: pid=43825
> key=1, fd=4
> _fdserver.c:468:handle_request():storing {ctx=1, key=1}->fd=6
> odp_pool.c:108:odp_pool_init_global():
> Pool init global
> odp_pool.c:109:odp_pool_init_global():  odp_buffer_hdr_t size 256
> odp_pool.c:110:odp_pool_init_global():  odp_packet_hdr_t size 576
> odp_pool.c:111:odp_pool_init_global():
> odp_queue_basic.c:78:queue_init_global():Queue init ...
> _fdserver.c:284:_odp_fdserver_register_fd():FD client register: pid=43825
> key=2, fd=5
> _fdserver.c:468:handle_request():storing {ctx=1, key=2}->fd=7
> odp_queue_lf.c:272:queue_lf_init_global():
> Lock-free queue init
> odp_queue_lf.c:273:queue_lf_init_global():  u128 lock-free: 1
>
> _fdserver.c:284:_odp_fdserver_register_fd():FD client register: pid=43825
> key=3, fd=6
> _fdserver.c:468:handle_request():storing {ctx=1, key=3}->fd=8
> odp_queue_basic.c:103:queue_init_global():done
> odp_queue_basic.c:104:queue_init_global():Queue init global
> odp_queue_basic.c:106:queue_init_global():  struct queue_entry_s size 256
> odp_queue_basic.c:108:queue_init_global():  queue_entry_t size256
> odp_queue_basic.c:109:queue_init_global():
> Using scheduler 'basic'
> odp_schedule_basic.c:305:schedule_init_global():Schedule init ...
> _fdserver.c:284:_odp_fdserver_register_fd():FD client register: pid=43825
> key=4, fd=7
> _fdserver.c:468:handle_request():storing {ctx=1, key=4}->fd=9
> odp_schedule_basic.c:365:schedule_init_global():done
> _fdserver.c:284:_odp_fdserver_register_fd():FD client register: pid=43825
> key=5, fd=8
> _fdserver.c:468:handle_request():storing {ctx=1, key=5}->fd=10
> PKTIO: initialized loop interface.
> PKTIO: initialized dpdk pktio, use export ODP_PKTIO_DISABLE_DPDK=1 to
> disable.
> PKTIO: initialized pcap interface.
> PKTIO: initialized ipc interface.
> PKTIO: initialized null interface.
> PKTIO: initialized socket mmap, use export ODP_PKTIO_DISABLE_SOCKET_MMAP=1
> to disable.
> PKTIO: initialized socket mmsg,use export ODP_PKTIO_DISABLE_SOCKET_MMSG=1
> to 

Re: [lng-odp] issues with usage of mellanox 100G NICs with ODP & ODP-DPDK

[gyanesh patra]

Hi,
I am coming back with the same issue here. As the odp-dpdk is updated to
the recent DPDK version inline with the ODP code base, now both ODP repo
and odp-dpdk repos stops working with mellanox interfaces. Please let me
know if this is a known issue. I tried to run with master branch,
caterpillar branch with and without 'abi-compat'. The ODP and odp-dpdk code
compilation process is kept equal. The dpdk is compiled with mlx flags and
working properly.

Here is the configuratin details and the errors i get while trying to run
./test/performance/odp_l2fwd use case with odp github code base (master
branch and has the v1.18.0.0 tag).

<>
ubuntu@ubuntu:~# uname -a
Linux ubuntu 4.4.0-112-generic #135-Ubuntu SMP Fri Jan 19 11:48:36 UTC 2018
x86_64 x86_64 x86_64 GNU/Linux

ubuntu@ubuntu:/home/gyanesh/dpdk-17.11# ./usertools/dpdk-devbind.py -s

Network devices using DPDK-compatible driver



Network devices using kernel driver
===
:81:00.0 'MT27700 Family [ConnectX-4] 1013' if=enp129s0f0 drv=mlx5_core
unused=
:81:00.1 'MT27700 Family [ConnectX-4] 1013' if=enp129s0f1 drv=mlx5_core
unused=

<>
ubuntu@ubuntu:/home/gyanesh/odp# git branch -a
  caterpillar
* master
  remotes/origin/HEAD -> origin/master
  remotes/origin/api-next
  remotes/origin/caterpillar

ubuntu@ubuntu:/home/gyanesh/odp# ./bootstrap
ubuntu@ubuntu:/home/gyanesh/odp# ./configure
--prefix=/home/gyanesh/odp/build
--with-dpdk-path=/home/gyanesh/dpdk-17.11/x86_64-native-linuxapp-gcc/
--enable-dpdk-zero-copy --disable-abi-compat --enable-debug-print
--enable-debug --enable-helper-linux --disable-doxygen-doc
--enable-helper-debug-print --enable-test-example --enable-test-helper
--enable-test-perf --enable-test-vald LDFLAGS="-libverbs -lmlx5"

ubuntu@ubuntu:/home/gyanesh/odp# make
ubuntu@ubuntu:/home/gyanesh/odp# cd test/performance

<< performance testcase >>
ubuntu@ubuntu:/home/gyanesh/odp/test/performance# export
ODP_PLATFORM_PARAMS="-n 4 -m 15240"
ubuntu@ubuntu:/home/gyanesh/odp/test/performance# ./odp_l2fwd -i 0,1 -c 2
HW time counter freq: 240786 hz

odp_system_info.c:100:default_huge_page_size():defaut hp size is 1048576 kB
odp_system_info.c:100:default_huge_page_size():defaut hp size is 1048576 kB
_ishm.c:1468:_odp_ishm_init_global():ishm: using dir /dev/shm
_ishm.c:1484:_odp_ishm_init_global():Huge pages mount point is:
/dev/hugepages
_ishmphy.c:65:_odp_ishmphy_book_va():VA Reserved: 0x7f07dd70b000,
len=0x6000
_fdserver.c:284:_odp_fdserver_register_fd():FD client register: pid=43825
key=0, fd=3
_fdserver.c:468:handle_request():storing {ctx=1, key=0}->fd=5
_fdserver.c:284:_odp_fdserver_register_fd():FD client register: pid=43825
key=1, fd=4
_fdserver.c:468:handle_request():storing {ctx=1, key=1}->fd=6
odp_pool.c:108:odp_pool_init_global():
Pool init global
odp_pool.c:109:odp_pool_init_global():  odp_buffer_hdr_t size 256
odp_pool.c:110:odp_pool_init_global():  odp_packet_hdr_t size 576
odp_pool.c:111:odp_pool_init_global():
odp_queue_basic.c:78:queue_init_global():Queue init ...
_fdserver.c:284:_odp_fdserver_register_fd():FD client register: pid=43825
key=2, fd=5
_fdserver.c:468:handle_request():storing {ctx=1, key=2}->fd=7
odp_queue_lf.c:272:queue_lf_init_global():
Lock-free queue init
odp_queue_lf.c:273:queue_lf_init_global():  u128 lock-free: 1

_fdserver.c:284:_odp_fdserver_register_fd():FD client register: pid=43825
key=3, fd=6
_fdserver.c:468:handle_request():storing {ctx=1, key=3}->fd=8
odp_queue_basic.c:103:queue_init_global():done
odp_queue_basic.c:104:queue_init_global():Queue init global
odp_queue_basic.c:106:queue_init_global():  struct queue_entry_s size 256
odp_queue_basic.c:108:queue_init_global():  queue_entry_t size256
odp_queue_basic.c:109:queue_init_global():
Using scheduler 'basic'
odp_schedule_basic.c:305:schedule_init_global():Schedule init ...
_fdserver.c:284:_odp_fdserver_register_fd():FD client register: pid=43825
key=4, fd=7
_fdserver.c:468:handle_request():storing {ctx=1, key=4}->fd=9
odp_schedule_basic.c:365:schedule_init_global():done
_fdserver.c:284:_odp_fdserver_register_fd():FD client register: pid=43825
key=5, fd=8
_fdserver.c:468:handle_request():storing {ctx=1, key=5}->fd=10
PKTIO: initialized loop interface.
PKTIO: initialized dpdk pktio, use export ODP_PKTIO_DISABLE_DPDK=1 to
disable.
PKTIO: initialized pcap interface.
PKTIO: initialized ipc interface.
PKTIO: initialized null interface.
PKTIO: initialized socket mmap, use export ODP_PKTIO_DISABLE_SOCKET_MMAP=1
to disable.
PKTIO: initialized socket mmsg,use export ODP_PKTIO_DISABLE_SOCKET_MMSG=1
to disable.
odp_timer.c:1253:odp_timer_init_global():Using lock-less timer
implementation
_fdserver.c:284:_odp_fdserver_register_fd():FD client register: pid=43825
key=6, fd=9
_fdserver.c:468:handle_request():storing {ctx=1, key=6}->fd=11
_fdserver.c:284:_odp_fdserver_register_fd():FD client register: pid=43825
key=7, fd=10
_fdserver.c:468:handle_request():storing {ctx=1, 

Re: [lng-odp] issues with usage of mellanox 100G NICs with ODP & ODP-DPDK

[gyanesh patra]

With my initial testing, ODP-DPDK is working perfectly with mellanox
drivers.
Thank you

P Gyanesh Kumar Patra

On Thu, Nov 9, 2017 at 2:18 PM, Maxim Uvarov 
wrote:

> Nice to see it working. I think we did not yet tested it with Mellanox
> drivers.
>
> For linux-generic refer to .travis.yaml or ./scripts/build-pktio-dpdk
> scripts. Also all required steps are in README.
>
> Maxim.
>
> On 11/09/17 14:47, Elo, Matias (Nokia - FI/Espoo) wrote:
> > Hi Gyanesh,
> >
> > Pretty much the same steps should also work with odp linux-generic. The
> main difference is configure script. With linux-generic you use
> '--with-dpdk-path=' option and optionally
> --enable-dpdk-zero-copy flag. The supported dpdk  version is v17.08.
> >
> > -Matias
> >
> >> On 9 Nov 2017, at 10:34, gyanesh patra 
> wrote:
> >>
> >> Hi Maxim,
> >> Thanks for the help. I managed to figure out the configuration error
> and it
> >> works fine for "ODP-DPDK". The MLX5 pmd was not included properly.
> >>
> >> But regarding "ODP" repo (not odp-dpdk), do i need to follow any steps
> to
> >> be able to use MLX ???
> >>
> >>
> >> P Gyanesh Kumar Patra
> >>
> >> On Wed, Nov 8, 2017 at 7:56 PM, Maxim Uvarov 
> >> wrote:
> >>
> >>> On 11/08/17 19:32, gyanesh patra wrote:
>  I am not sure what you mean. Can you please elaborate?
> 
>  As i mentioned before I am able to run dpdk examples. Hence the
> drivers
>  are available and working fine.
>  I configured ODP & ODP-DPDK with "LDFLAGS=-libverbs" and compiled to
>  work with mellanox. I followed the same while compiling dpdk too.
> 
>  Is there anything i am missing?
> 
>  P Gyanesh Kumar Patra
> >>>
> >>>
> >>> in general if CONFIG_RTE_LIBRTE_MLX5_PMD=y was specified then it has to
> >>> work. I think we did test only with ixgbe. But in general it's common
> code.
> >>>
> >>> "Unable to init any I/O type." means it it called all open for all
> pktio
> >>> in list here:
> >>> ./platform/linux-generic/pktio/io_ops.c
> >>>
> >>> and setup_pkt_dpdk() failed for some reason.
> >>>
> >>> I do not like allocations errors in your log.
> >>>
> >>> Try to compile ODP with --enable-debug-print --enable-debug it will
> make
> >>> ODP_DBG() macro work and it will be visible why it does not opens
> pktio.
> >>>
> >>> Maxim
> >>>
> >>>
> 
>  On Wed, Nov 8, 2017 at 5:22 PM, Maxim Uvarov   > wrote:
> 
> is Mellanox pmd compiled in?
> 
> Maxim.
> 
> On 11/08/17 17:58, gyanesh patra wrote:
> > Hi,
> > I am trying to run ODP & ODP-DPDK examples on our server with
> mellanox 100G
> > NICs. I am using the odp_l2fwd example. While running the example,
> I am
> > facing some issues.
> > -> When I run "ODP" example using the if names given by kernel as
> > arguments, I am not getting enough throughput.(the value is very
> >>> low)
> > -> And when I try "ODP-DPDK" example using port ID as "0,1", it
> >>> can't
> > create pktio. Whereas I am able to run the examples from "DPDK"
> > repo with portID "0,1" for the same mellanox NICs. I tried running
> with
> > "81:00.0,81:00.1" and also with if-names too without any success.
> Adding
> > the whitelist using ODP_PLATFORM_PARAMS doesn't help either.
> >
> > Am I missing any steps to use mellanox NICs? OR is there a
> different method
> > to specify the device details to create pktio?
> > I am providing the output of "odp_l2fwd" examples for ODP and
> >>> ODP-DPDK
> > repository here.
> >
> > The NICs being used:
> >
> > :81:00.0 'MT27700 Family [ConnectX-4]' if=enp129s0f0
> >>> drv=mlx5_core
> > unused=
> > :81:00.1 'MT27700 Family [ConnectX-4]' if=enp129s0f1
> >>> drv=mlx5_core
> > unused=
> >
> > ODP l2fwd example run details:
> > --
> > root@ubuntu:/home/ubuntu/odp/test/performance# ./odp_l2fwd -i
> > enp129s0f0,enp129s0f1
> > HW time counter freq: 239886 
> <(239)%20999-9886> hz
> >
> > _ishmphy.c:152:_odp_ishmphy_map():mmap failed:Cannot allocate
> >>> memory
> > _ishm.c:880:_odp_ishm_reserve():No huge pages, fall back to normal
> pages.
> > check: /proc/sys/vm/nr_hugepages.
> > _ishmphy.c:152:_odp_ishmphy_map():mmap failed:Cannot allocate
> >>> memory
> > _ishmphy.c:152:_odp_ishmphy_map():mmap failed:Cannot allocate
> >>> memory
> > _ishmphy.c:152:_odp_ishmphy_map():mmap failed:Cannot allocate
> >>> memory
> > PKTIO: initialized loop interface.
> > PKTIO: initialized pcap interface.
> > PKTIO: initialized ipc interface.
> > PKTIO: initialized socket mmap, use export
> ODP_PKTIO_DISABLE_SOCKET_MMAP=1
> > to disable.
> > PKTIO: initialized socket mmsg,use export
> 

Re: [lng-odp] issues with usage of mellanox 100G NICs with ODP & ODP-DPDK

[gyanesh patra]

This is good news. I was wondering if it was in the pipeline. Thank you

P Gyanesh Kumar Patra

On Thu, Nov 9, 2017 at 7:25 PM, Francois Ozog 
wrote:

> ODP2.0 should allow ODP to leverage directly libiverbs from a native ODP
> pktio without DPDK layer.
> Mellanox has created a userland framework based on libiverbs while we try
> to promote an extension of Mediated Device (vfio-mdev).
>
> FF
>
> On 9 November 2017 at 14:18, Maxim Uvarov  wrote:
>
>> Nice to see it working. I think we did not yet tested it with Mellanox
>> drivers.
>>
>> For linux-generic refer to .travis.yaml or ./scripts/build-pktio-dpdk
>> scripts. Also all required steps are in README.
>>
>> Maxim.
>>
>> On 11/09/17 14:47, Elo, Matias (Nokia - FI/Espoo) wrote:
>> > Hi Gyanesh,
>> >
>> > Pretty much the same steps should also work with odp linux-generic. The
>> main difference is configure script. With linux-generic you use
>> '--with-dpdk-path=' option and optionally
>> --enable-dpdk-zero-copy flag. The supported dpdk  version is v17.08.
>> >
>> > -Matias
>> >
>> >> On 9 Nov 2017, at 10:34, gyanesh patra 
>> wrote:
>> >>
>> >> Hi Maxim,
>> >> Thanks for the help. I managed to figure out the configuration error
>> and it
>> >> works fine for "ODP-DPDK". The MLX5 pmd was not included properly.
>> >>
>> >> But regarding "ODP" repo (not odp-dpdk), do i need to follow any steps
>> to
>> >> be able to use MLX ???
>> >>
>> >>
>> >> P Gyanesh Kumar Patra
>> >>
>> >> On Wed, Nov 8, 2017 at 7:56 PM, Maxim Uvarov 
>> >> wrote:
>> >>
>> >>> On 11/08/17 19:32, gyanesh patra wrote:
>>  I am not sure what you mean. Can you please elaborate?
>> 
>>  As i mentioned before I am able to run dpdk examples. Hence the
>> drivers
>>  are available and working fine.
>>  I configured ODP & ODP-DPDK with "LDFLAGS=-libverbs" and compiled to
>>  work with mellanox. I followed the same while compiling dpdk too.
>> 
>>  Is there anything i am missing?
>> 
>>  P Gyanesh Kumar Patra
>> >>>
>> >>>
>> >>> in general if CONFIG_RTE_LIBRTE_MLX5_PMD=y was specified then it has
>> to
>> >>> work. I think we did test only with ixgbe. But in general it's common
>> code.
>> >>>
>> >>> "Unable to init any I/O type." means it it called all open for all
>> pktio
>> >>> in list here:
>> >>> ./platform/linux-generic/pktio/io_ops.c
>> >>>
>> >>> and setup_pkt_dpdk() failed for some reason.
>> >>>
>> >>> I do not like allocations errors in your log.
>> >>>
>> >>> Try to compile ODP with --enable-debug-print --enable-debug it will
>> make
>> >>> ODP_DBG() macro work and it will be visible why it does not opens
>> pktio.
>> >>>
>> >>> Maxim
>> >>>
>> >>>
>> 
>>  On Wed, Nov 8, 2017 at 5:22 PM, Maxim Uvarov <
>> maxim.uva...@linaro.org
>>  > wrote:
>> 
>> is Mellanox pmd compiled in?
>> 
>> Maxim.
>> 
>> On 11/08/17 17:58, gyanesh patra wrote:
>> > Hi,
>> > I am trying to run ODP & ODP-DPDK examples on our server with
>> mellanox 100G
>> > NICs. I am using the odp_l2fwd example. While running the example,
>> I am
>> > facing some issues.
>> > -> When I run "ODP" example using the if names given by kernel as
>> > arguments, I am not getting enough throughput.(the value is very
>> >>> low)
>> > -> And when I try "ODP-DPDK" example using port ID as "0,1", it
>> >>> can't
>> > create pktio. Whereas I am able to run the examples from "DPDK"
>> > repo with portID "0,1" for the same mellanox NICs. I tried running
>> with
>> > "81:00.0,81:00.1" and also with if-names too without any success.
>> Adding
>> > the whitelist using ODP_PLATFORM_PARAMS doesn't help either.
>> >
>> > Am I missing any steps to use mellanox NICs? OR is there a
>> different method
>> > to specify the device details to create pktio?
>> > I am providing the output of "odp_l2fwd" examples for ODP and
>> >>> ODP-DPDK
>> > repository here.
>> >
>> > The NICs being used:
>> >
>> > :81:00.0 'MT27700 Family [ConnectX-4]' if=enp129s0f0
>> >>> drv=mlx5_core
>> > unused=
>> > :81:00.1 'MT27700 Family [ConnectX-4]' if=enp129s0f1
>> >>> drv=mlx5_core
>> > unused=
>> >
>> > ODP l2fwd example run details:
>> > --
>> > root@ubuntu:/home/ubuntu/odp/test/performance# ./odp_l2fwd -i
>> > enp129s0f0,enp129s0f1
>> > HW time counter freq: 239886 <(239)%20999-9886> > <(239)%20999-9886>>
>> <(239)%20999-9886> hz
>> >
>> > _ishmphy.c:152:_odp_ishmphy_map():mmap failed:Cannot allocate
>> >>> memory
>> > _ishm.c:880:_odp_ishm_reserve():No huge pages, fall back to normal
>> pages.
>> > check: /proc/sys/vm/nr_hugepages.
>> > _ishmphy.c:152:_odp_ishmphy_map():mmap failed:Cannot allocate
>> >>> memory
>> 

Re: [lng-odp] issues with usage of mellanox 100G NICs with ODP & ODP-DPDK

[Francois Ozog]

ODP2.0 should allow ODP to leverage directly libiverbs from a native ODP
pktio without DPDK layer.
Mellanox has created a userland framework based on libiverbs while we try
to promote an extension of Mediated Device (vfio-mdev).

FF

On 9 November 2017 at 14:18, Maxim Uvarov  wrote:

> Nice to see it working. I think we did not yet tested it with Mellanox
> drivers.
>
> For linux-generic refer to .travis.yaml or ./scripts/build-pktio-dpdk
> scripts. Also all required steps are in README.
>
> Maxim.
>
> On 11/09/17 14:47, Elo, Matias (Nokia - FI/Espoo) wrote:
> > Hi Gyanesh,
> >
> > Pretty much the same steps should also work with odp linux-generic. The
> main difference is configure script. With linux-generic you use
> '--with-dpdk-path=' option and optionally
> --enable-dpdk-zero-copy flag. The supported dpdk  version is v17.08.
> >
> > -Matias
> >
> >> On 9 Nov 2017, at 10:34, gyanesh patra 
> wrote:
> >>
> >> Hi Maxim,
> >> Thanks for the help. I managed to figure out the configuration error
> and it
> >> works fine for "ODP-DPDK". The MLX5 pmd was not included properly.
> >>
> >> But regarding "ODP" repo (not odp-dpdk), do i need to follow any steps
> to
> >> be able to use MLX ???
> >>
> >>
> >> P Gyanesh Kumar Patra
> >>
> >> On Wed, Nov 8, 2017 at 7:56 PM, Maxim Uvarov 
> >> wrote:
> >>
> >>> On 11/08/17 19:32, gyanesh patra wrote:
>  I am not sure what you mean. Can you please elaborate?
> 
>  As i mentioned before I am able to run dpdk examples. Hence the
> drivers
>  are available and working fine.
>  I configured ODP & ODP-DPDK with "LDFLAGS=-libverbs" and compiled to
>  work with mellanox. I followed the same while compiling dpdk too.
> 
>  Is there anything i am missing?
> 
>  P Gyanesh Kumar Patra
> >>>
> >>>
> >>> in general if CONFIG_RTE_LIBRTE_MLX5_PMD=y was specified then it has to
> >>> work. I think we did test only with ixgbe. But in general it's common
> code.
> >>>
> >>> "Unable to init any I/O type." means it it called all open for all
> pktio
> >>> in list here:
> >>> ./platform/linux-generic/pktio/io_ops.c
> >>>
> >>> and setup_pkt_dpdk() failed for some reason.
> >>>
> >>> I do not like allocations errors in your log.
> >>>
> >>> Try to compile ODP with --enable-debug-print --enable-debug it will
> make
> >>> ODP_DBG() macro work and it will be visible why it does not opens
> pktio.
> >>>
> >>> Maxim
> >>>
> >>>
> 
>  On Wed, Nov 8, 2017 at 5:22 PM, Maxim Uvarov   > wrote:
> 
> is Mellanox pmd compiled in?
> 
> Maxim.
> 
> On 11/08/17 17:58, gyanesh patra wrote:
> > Hi,
> > I am trying to run ODP & ODP-DPDK examples on our server with
> mellanox 100G
> > NICs. I am using the odp_l2fwd example. While running the example,
> I am
> > facing some issues.
> > -> When I run "ODP" example using the if names given by kernel as
> > arguments, I am not getting enough throughput.(the value is very
> >>> low)
> > -> And when I try "ODP-DPDK" example using port ID as "0,1", it
> >>> can't
> > create pktio. Whereas I am able to run the examples from "DPDK"
> > repo with portID "0,1" for the same mellanox NICs. I tried running
> with
> > "81:00.0,81:00.1" and also with if-names too without any success.
> Adding
> > the whitelist using ODP_PLATFORM_PARAMS doesn't help either.
> >
> > Am I missing any steps to use mellanox NICs? OR is there a
> different method
> > to specify the device details to create pktio?
> > I am providing the output of "odp_l2fwd" examples for ODP and
> >>> ODP-DPDK
> > repository here.
> >
> > The NICs being used:
> >
> > :81:00.0 'MT27700 Family [ConnectX-4]' if=enp129s0f0
> >>> drv=mlx5_core
> > unused=
> > :81:00.1 'MT27700 Family [ConnectX-4]' if=enp129s0f1
> >>> drv=mlx5_core
> > unused=
> >
> > ODP l2fwd example run details:
> > --
> > root@ubuntu:/home/ubuntu/odp/test/performance# ./odp_l2fwd -i
> > enp129s0f0,enp129s0f1
> > HW time counter freq: 239886 
> <(239)%20999-9886> hz
> >
> > _ishmphy.c:152:_odp_ishmphy_map():mmap failed:Cannot allocate
> >>> memory
> > _ishm.c:880:_odp_ishm_reserve():No huge pages, fall back to normal
> pages.
> > check: /proc/sys/vm/nr_hugepages.
> > _ishmphy.c:152:_odp_ishmphy_map():mmap failed:Cannot allocate
> >>> memory
> > _ishmphy.c:152:_odp_ishmphy_map():mmap failed:Cannot allocate
> >>> memory
> > _ishmphy.c:152:_odp_ishmphy_map():mmap failed:Cannot allocate
> >>> memory
> > PKTIO: initialized loop interface.
> > PKTIO: initialized pcap interface.
> > PKTIO: initialized ipc interface.
> > PKTIO: initialized socket mmap, use export
> 

Re: [lng-odp] issues with usage of mellanox 100G NICs with ODP & ODP-DPDK

[Maxim Uvarov]

Nice to see it working. I think we did not yet tested it with Mellanox
drivers.

For linux-generic refer to .travis.yaml or ./scripts/build-pktio-dpdk
scripts. Also all required steps are in README.

Maxim.

On 11/09/17 14:47, Elo, Matias (Nokia - FI/Espoo) wrote:
> Hi Gyanesh,
> 
> Pretty much the same steps should also work with odp linux-generic. The main 
> difference is configure script. With linux-generic you use 
> '--with-dpdk-path=' option and optionally --enable-dpdk-zero-copy 
> flag. The supported dpdk  version is v17.08.
> 
> -Matias
> 
>> On 9 Nov 2017, at 10:34, gyanesh patra  wrote:
>>
>> Hi Maxim,
>> Thanks for the help. I managed to figure out the configuration error and it
>> works fine for "ODP-DPDK". The MLX5 pmd was not included properly.
>>
>> But regarding "ODP" repo (not odp-dpdk), do i need to follow any steps to
>> be able to use MLX ???
>>
>>
>> P Gyanesh Kumar Patra
>>
>> On Wed, Nov 8, 2017 at 7:56 PM, Maxim Uvarov 
>> wrote:
>>
>>> On 11/08/17 19:32, gyanesh patra wrote:
 I am not sure what you mean. Can you please elaborate?

 As i mentioned before I am able to run dpdk examples. Hence the drivers
 are available and working fine.
 I configured ODP & ODP-DPDK with "LDFLAGS=-libverbs" and compiled to
 work with mellanox. I followed the same while compiling dpdk too.

 Is there anything i am missing?

 P Gyanesh Kumar Patra
>>>
>>>
>>> in general if CONFIG_RTE_LIBRTE_MLX5_PMD=y was specified then it has to
>>> work. I think we did test only with ixgbe. But in general it's common code.
>>>
>>> "Unable to init any I/O type." means it it called all open for all pktio
>>> in list here:
>>> ./platform/linux-generic/pktio/io_ops.c
>>>
>>> and setup_pkt_dpdk() failed for some reason.
>>>
>>> I do not like allocations errors in your log.
>>>
>>> Try to compile ODP with --enable-debug-print --enable-debug it will make
>>> ODP_DBG() macro work and it will be visible why it does not opens pktio.
>>>
>>> Maxim
>>>
>>>

 On Wed, Nov 8, 2017 at 5:22 PM, Maxim Uvarov > wrote:

is Mellanox pmd compiled in?

Maxim.

On 11/08/17 17:58, gyanesh patra wrote:
> Hi,
> I am trying to run ODP & ODP-DPDK examples on our server with
mellanox 100G
> NICs. I am using the odp_l2fwd example. While running the example,
I am
> facing some issues.
> -> When I run "ODP" example using the if names given by kernel as
> arguments, I am not getting enough throughput.(the value is very
>>> low)
> -> And when I try "ODP-DPDK" example using port ID as "0,1", it
>>> can't
> create pktio. Whereas I am able to run the examples from "DPDK"
> repo with portID "0,1" for the same mellanox NICs. I tried running
with
> "81:00.0,81:00.1" and also with if-names too without any success.
Adding
> the whitelist using ODP_PLATFORM_PARAMS doesn't help either.
>
> Am I missing any steps to use mellanox NICs? OR is there a
different method
> to specify the device details to create pktio?
> I am providing the output of "odp_l2fwd" examples for ODP and
>>> ODP-DPDK
> repository here.
>
> The NICs being used:
>
> :81:00.0 'MT27700 Family [ConnectX-4]' if=enp129s0f0
>>> drv=mlx5_core
> unused=
> :81:00.1 'MT27700 Family [ConnectX-4]' if=enp129s0f1
>>> drv=mlx5_core
> unused=
>
> ODP l2fwd example run details:
> --
> root@ubuntu:/home/ubuntu/odp/test/performance# ./odp_l2fwd -i
> enp129s0f0,enp129s0f1
> HW time counter freq: 239886 
<(239)%20999-9886> hz
>
> _ishmphy.c:152:_odp_ishmphy_map():mmap failed:Cannot allocate
>>> memory
> _ishm.c:880:_odp_ishm_reserve():No huge pages, fall back to normal
pages.
> check: /proc/sys/vm/nr_hugepages.
> _ishmphy.c:152:_odp_ishmphy_map():mmap failed:Cannot allocate
>>> memory
> _ishmphy.c:152:_odp_ishmphy_map():mmap failed:Cannot allocate
>>> memory
> _ishmphy.c:152:_odp_ishmphy_map():mmap failed:Cannot allocate
>>> memory
> PKTIO: initialized loop interface.
> PKTIO: initialized pcap interface.
> PKTIO: initialized ipc interface.
> PKTIO: initialized socket mmap, use export
ODP_PKTIO_DISABLE_SOCKET_MMAP=1
> to disable.
> PKTIO: initialized socket mmsg,use export
ODP_PKTIO_DISABLE_SOCKET_MMSG=1
> to disable.
> _ishmphy.c:152:_odp_ishmphy_map():mmap failed:Cannot allocate
>>> memory
> _ishmphy.c:152:_odp_ishmphy_map():mmap failed:Cannot allocate
>>> memory
> _ishmphy.c:152:_odp_ishmphy_map():mmap failed:Cannot allocate
>>> memory
> _ishmphy.c:152:_odp_ishmphy_map():mmap failed:Cannot allocate
>>> memory
>
> ODP system info
> ---
> ODP API version: 1.15.0
> ODP impl name:   "odp-linux"

Re: [lng-odp] issues with usage of mellanox 100G NICs with ODP & ODP-DPDK

[Elo, Matias (Nokia - FI/Espoo)]

Hi Gyanesh,

Pretty much the same steps should also work with odp linux-generic. The main 
difference is configure script. With linux-generic you use 
'--with-dpdk-path=' option and optionally --enable-dpdk-zero-copy 
flag. The supported dpdk  version is v17.08.

-Matias

> On 9 Nov 2017, at 10:34, gyanesh patra  wrote:
> 
> Hi Maxim,
> Thanks for the help. I managed to figure out the configuration error and it
> works fine for "ODP-DPDK". The MLX5 pmd was not included properly.
> 
> But regarding "ODP" repo (not odp-dpdk), do i need to follow any steps to
> be able to use MLX ???
> 
> 
> P Gyanesh Kumar Patra
> 
> On Wed, Nov 8, 2017 at 7:56 PM, Maxim Uvarov 
> wrote:
> 
>> On 11/08/17 19:32, gyanesh patra wrote:
>>> I am not sure what you mean. Can you please elaborate?
>>> 
>>> As i mentioned before I am able to run dpdk examples. Hence the drivers
>>> are available and working fine.
>>> I configured ODP & ODP-DPDK with "LDFLAGS=-libverbs" and compiled to
>>> work with mellanox. I followed the same while compiling dpdk too.
>>> 
>>> Is there anything i am missing?
>>> 
>>> P Gyanesh Kumar Patra
>> 
>> 
>> in general if CONFIG_RTE_LIBRTE_MLX5_PMD=y was specified then it has to
>> work. I think we did test only with ixgbe. But in general it's common code.
>> 
>> "Unable to init any I/O type." means it it called all open for all pktio
>> in list here:
>> ./platform/linux-generic/pktio/io_ops.c
>> 
>> and setup_pkt_dpdk() failed for some reason.
>> 
>> I do not like allocations errors in your log.
>> 
>> Try to compile ODP with --enable-debug-print --enable-debug it will make
>> ODP_DBG() macro work and it will be visible why it does not opens pktio.
>> 
>> Maxim
>> 
>> 
>>> 
>>> On Wed, Nov 8, 2017 at 5:22 PM, Maxim Uvarov >> > wrote:
>>> 
>>>is Mellanox pmd compiled in?
>>> 
>>>Maxim.
>>> 
>>>On 11/08/17 17:58, gyanesh patra wrote:
 Hi,
 I am trying to run ODP & ODP-DPDK examples on our server with
>>>mellanox 100G
 NICs. I am using the odp_l2fwd example. While running the example,
>>>I am
 facing some issues.
 -> When I run "ODP" example using the if names given by kernel as
 arguments, I am not getting enough throughput.(the value is very
>> low)
 -> And when I try "ODP-DPDK" example using port ID as "0,1", it
>> can't
 create pktio. Whereas I am able to run the examples from "DPDK"
 repo with portID "0,1" for the same mellanox NICs. I tried running
>>>with
 "81:00.0,81:00.1" and also with if-names too without any success.
>>>Adding
 the whitelist using ODP_PLATFORM_PARAMS doesn't help either.
 
 Am I missing any steps to use mellanox NICs? OR is there a
>>>different method
 to specify the device details to create pktio?
 I am providing the output of "odp_l2fwd" examples for ODP and
>> ODP-DPDK
 repository here.
 
 The NICs being used:
 
 :81:00.0 'MT27700 Family [ConnectX-4]' if=enp129s0f0
>> drv=mlx5_core
 unused=
 :81:00.1 'MT27700 Family [ConnectX-4]' if=enp129s0f1
>> drv=mlx5_core
 unused=
 
 ODP l2fwd example run details:
 --
 root@ubuntu:/home/ubuntu/odp/test/performance# ./odp_l2fwd -i
 enp129s0f0,enp129s0f1
 HW time counter freq: 239886 
>>><(239)%20999-9886> hz
 
 _ishmphy.c:152:_odp_ishmphy_map():mmap failed:Cannot allocate
>> memory
 _ishm.c:880:_odp_ishm_reserve():No huge pages, fall back to normal
>>>pages.
 check: /proc/sys/vm/nr_hugepages.
 _ishmphy.c:152:_odp_ishmphy_map():mmap failed:Cannot allocate
>> memory
 _ishmphy.c:152:_odp_ishmphy_map():mmap failed:Cannot allocate
>> memory
 _ishmphy.c:152:_odp_ishmphy_map():mmap failed:Cannot allocate
>> memory
 PKTIO: initialized loop interface.
 PKTIO: initialized pcap interface.
 PKTIO: initialized ipc interface.
 PKTIO: initialized socket mmap, use export
>>>ODP_PKTIO_DISABLE_SOCKET_MMAP=1
 to disable.
 PKTIO: initialized socket mmsg,use export
>>>ODP_PKTIO_DISABLE_SOCKET_MMSG=1
 to disable.
 _ishmphy.c:152:_odp_ishmphy_map():mmap failed:Cannot allocate
>> memory
 _ishmphy.c:152:_odp_ishmphy_map():mmap failed:Cannot allocate
>> memory
 _ishmphy.c:152:_odp_ishmphy_map():mmap failed:Cannot allocate
>> memory
 _ishmphy.c:152:_odp_ishmphy_map():mmap failed:Cannot allocate
>> memory
 
 ODP system info
 ---
 ODP API version: 1.15.0
 ODP impl name:   "odp-linux"
 CPU model:   Intel(R) Xeon(R) CPU E5-2680 v4
 CPU freq (hz):   33
 Cache line size: 64
 CPU count:   56
 
 
 CPU features supported:
 SSE3 PCLMULQDQ DTES64 MONITOR DS_CPL VMX SMX EIST TM2 SSSE3 FMA
>>>CMPXCHG16B
 XTPR PDCM PCID DCA SSE4_1 SSE4_2 X2APIC MOVBE POPCNT TSC_DEADLINE
>>>AES XSAVE
 OSXSAVE AVX F16C RDRAND 

Re: [lng-odp] issues with usage of mellanox 100G NICs with ODP & ODP-DPDK

[gyanesh patra]

Hi Maxim,
Thanks for the help. I managed to figure out the configuration error and it
works fine for "ODP-DPDK". The MLX5 pmd was not included properly.

But regarding "ODP" repo (not odp-dpdk), do i need to follow any steps to
be able to use MLX ???


P Gyanesh Kumar Patra

On Wed, Nov 8, 2017 at 7:56 PM, Maxim Uvarov 
wrote:

> On 11/08/17 19:32, gyanesh patra wrote:
> > I am not sure what you mean. Can you please elaborate?
> >
> > As i mentioned before I am able to run dpdk examples. Hence the drivers
> > are available and working fine.
> > I configured ODP & ODP-DPDK with "LDFLAGS=-libverbs" and compiled to
> > work with mellanox. I followed the same while compiling dpdk too.
> >
> > Is there anything i am missing?
> >
> > P Gyanesh Kumar Patra
>
>
> in general if CONFIG_RTE_LIBRTE_MLX5_PMD=y was specified then it has to
> work. I think we did test only with ixgbe. But in general it's common code.
>
> "Unable to init any I/O type." means it it called all open for all pktio
> in list here:
> ./platform/linux-generic/pktio/io_ops.c
>
> and setup_pkt_dpdk() failed for some reason.
>
> I do not like allocations errors in your log.
>
> Try to compile ODP with --enable-debug-print --enable-debug it will make
> ODP_DBG() macro work and it will be visible why it does not opens pktio.
>
> Maxim
>
>
> >
> > On Wed, Nov 8, 2017 at 5:22 PM, Maxim Uvarov  > > wrote:
> >
> > is Mellanox pmd compiled in?
> >
> > Maxim.
> >
> > On 11/08/17 17:58, gyanesh patra wrote:
> > > Hi,
> > > I am trying to run ODP & ODP-DPDK examples on our server with
> > mellanox 100G
> > > NICs. I am using the odp_l2fwd example. While running the example,
> > I am
> > > facing some issues.
> > > -> When I run "ODP" example using the if names given by kernel as
> > > arguments, I am not getting enough throughput.(the value is very
> low)
> > > -> And when I try "ODP-DPDK" example using port ID as "0,1", it
> can't
> > > create pktio. Whereas I am able to run the examples from "DPDK"
> > > repo with portID "0,1" for the same mellanox NICs. I tried running
> > with
> > > "81:00.0,81:00.1" and also with if-names too without any success.
> > Adding
> > > the whitelist using ODP_PLATFORM_PARAMS doesn't help either.
> > >
> > > Am I missing any steps to use mellanox NICs? OR is there a
> > different method
> > > to specify the device details to create pktio?
> > > I am providing the output of "odp_l2fwd" examples for ODP and
> ODP-DPDK
> > > repository here.
> > >
> > > The NICs being used:
> > >
> > > :81:00.0 'MT27700 Family [ConnectX-4]' if=enp129s0f0
> drv=mlx5_core
> > > unused=
> > > :81:00.1 'MT27700 Family [ConnectX-4]' if=enp129s0f1
> drv=mlx5_core
> > > unused=
> > >
> > > ODP l2fwd example run details:
> > > --
> > > root@ubuntu:/home/ubuntu/odp/test/performance# ./odp_l2fwd -i
> > > enp129s0f0,enp129s0f1
> > > HW time counter freq: 239886 
> > <(239)%20999-9886> hz
> > >
> > > _ishmphy.c:152:_odp_ishmphy_map():mmap failed:Cannot allocate
> memory
> > > _ishm.c:880:_odp_ishm_reserve():No huge pages, fall back to normal
> > pages.
> > > check: /proc/sys/vm/nr_hugepages.
> > > _ishmphy.c:152:_odp_ishmphy_map():mmap failed:Cannot allocate
> memory
> > > _ishmphy.c:152:_odp_ishmphy_map():mmap failed:Cannot allocate
> memory
> > > _ishmphy.c:152:_odp_ishmphy_map():mmap failed:Cannot allocate
> memory
> > >  PKTIO: initialized loop interface.
> > >  PKTIO: initialized pcap interface.
> > >  PKTIO: initialized ipc interface.
> > >  PKTIO: initialized socket mmap, use export
> > ODP_PKTIO_DISABLE_SOCKET_MMAP=1
> > > to disable.
> > >  PKTIO: initialized socket mmsg,use export
> > ODP_PKTIO_DISABLE_SOCKET_MMSG=1
> > > to disable.
> > > _ishmphy.c:152:_odp_ishmphy_map():mmap failed:Cannot allocate
> memory
> > > _ishmphy.c:152:_odp_ishmphy_map():mmap failed:Cannot allocate
> memory
> > > _ishmphy.c:152:_odp_ishmphy_map():mmap failed:Cannot allocate
> memory
> > > _ishmphy.c:152:_odp_ishmphy_map():mmap failed:Cannot allocate
> memory
> > >
> > > ODP system info
> > > ---
> > > ODP API version: 1.15.0
> > > ODP impl name:   "odp-linux"
> > > CPU model:   Intel(R) Xeon(R) CPU E5-2680 v4
> > > CPU freq (hz):   33
> > > Cache line size: 64
> > > CPU count:   56
> > >
> > >
> > > CPU features supported:
> > > SSE3 PCLMULQDQ DTES64 MONITOR DS_CPL VMX SMX EIST TM2 SSSE3 FMA
> > CMPXCHG16B
> > > XTPR PDCM PCID DCA SSE4_1 SSE4_2 X2APIC MOVBE POPCNT TSC_DEADLINE
> > AES XSAVE
> > > OSXSAVE AVX F16C RDRAND FPU VME DE PSE TSC MSR PAE MCE CX8 APIC
> > SEP MTRR
> > > PGE MCA CMOV PAT PSE36 CLFSH DS ACPI 

Re: [lng-odp] issues with usage of mellanox 100G NICs with ODP & ODP-DPDK

[gyanesh patra]

- where to set CONFIG_RTE_LIBRTE_MLX5_PMD ? I believe in DPDK we set it
whilie compiling. How to set it for ODP??

- with mellanox NICs, it is not required to explicitly bind the interface
to dpdk. I can use portID "0,1"  with dpdk example. Is it the same way to
specify with ODP-DPDK l2fwd example too??


P Gyanesh Kumar Patra

On Wed, Nov 8, 2017 at 7:56 PM, Maxim Uvarov 
wrote:

> On 11/08/17 19:32, gyanesh patra wrote:
> > I am not sure what you mean. Can you please elaborate?
> >
> > As i mentioned before I am able to run dpdk examples. Hence the drivers
> > are available and working fine.
> > I configured ODP & ODP-DPDK with "LDFLAGS=-libverbs" and compiled to
> > work with mellanox. I followed the same while compiling dpdk too.
> >
> > Is there anything i am missing?
> >
> > P Gyanesh Kumar Patra
>
>
> in general if CONFIG_RTE_LIBRTE_MLX5_PMD=y was specified then it has to
> work. I think we did test only with ixgbe. But in general it's common code.
>
> "Unable to init any I/O type." means it it called all open for all pktio
> in list here:
> ./platform/linux-generic/pktio/io_ops.c
>
> and setup_pkt_dpdk() failed for some reason.
>
> I do not like allocations errors in your log.
>
> Try to compile ODP with --enable-debug-print --enable-debug it will make
> ODP_DBG() macro work and it will be visible why it does not opens pktio.
>
> Maxim
>
>
> >
> > On Wed, Nov 8, 2017 at 5:22 PM, Maxim Uvarov  > > wrote:
> >
> > is Mellanox pmd compiled in?
> >
> > Maxim.
> >
> > On 11/08/17 17:58, gyanesh patra wrote:
> > > Hi,
> > > I am trying to run ODP & ODP-DPDK examples on our server with
> > mellanox 100G
> > > NICs. I am using the odp_l2fwd example. While running the example,
> > I am
> > > facing some issues.
> > > -> When I run "ODP" example using the if names given by kernel as
> > > arguments, I am not getting enough throughput.(the value is very
> low)
> > > -> And when I try "ODP-DPDK" example using port ID as "0,1", it
> can't
> > > create pktio. Whereas I am able to run the examples from "DPDK"
> > > repo with portID "0,1" for the same mellanox NICs. I tried running
> > with
> > > "81:00.0,81:00.1" and also with if-names too without any success.
> > Adding
> > > the whitelist using ODP_PLATFORM_PARAMS doesn't help either.
> > >
> > > Am I missing any steps to use mellanox NICs? OR is there a
> > different method
> > > to specify the device details to create pktio?
> > > I am providing the output of "odp_l2fwd" examples for ODP and
> ODP-DPDK
> > > repository here.
> > >
> > > The NICs being used:
> > >
> > > :81:00.0 'MT27700 Family [ConnectX-4]' if=enp129s0f0
> drv=mlx5_core
> > > unused=
> > > :81:00.1 'MT27700 Family [ConnectX-4]' if=enp129s0f1
> drv=mlx5_core
> > > unused=
> > >
> > > ODP l2fwd example run details:
> > > --
> > > root@ubuntu:/home/ubuntu/odp/test/performance# ./odp_l2fwd -i
> > > enp129s0f0,enp129s0f1
> > > HW time counter freq: 239886 
> > <(239)%20999-9886> hz
> > >
> > > _ishmphy.c:152:_odp_ishmphy_map():mmap failed:Cannot allocate
> memory
> > > _ishm.c:880:_odp_ishm_reserve():No huge pages, fall back to normal
> > pages.
> > > check: /proc/sys/vm/nr_hugepages.
> > > _ishmphy.c:152:_odp_ishmphy_map():mmap failed:Cannot allocate
> memory
> > > _ishmphy.c:152:_odp_ishmphy_map():mmap failed:Cannot allocate
> memory
> > > _ishmphy.c:152:_odp_ishmphy_map():mmap failed:Cannot allocate
> memory
> > >  PKTIO: initialized loop interface.
> > >  PKTIO: initialized pcap interface.
> > >  PKTIO: initialized ipc interface.
> > >  PKTIO: initialized socket mmap, use export
> > ODP_PKTIO_DISABLE_SOCKET_MMAP=1
> > > to disable.
> > >  PKTIO: initialized socket mmsg,use export
> > ODP_PKTIO_DISABLE_SOCKET_MMSG=1
> > > to disable.
> > > _ishmphy.c:152:_odp_ishmphy_map():mmap failed:Cannot allocate
> memory
> > > _ishmphy.c:152:_odp_ishmphy_map():mmap failed:Cannot allocate
> memory
> > > _ishmphy.c:152:_odp_ishmphy_map():mmap failed:Cannot allocate
> memory
> > > _ishmphy.c:152:_odp_ishmphy_map():mmap failed:Cannot allocate
> memory
> > >
> > > ODP system info
> > > ---
> > > ODP API version: 1.15.0
> > > ODP impl name:   "odp-linux"
> > > CPU model:   Intel(R) Xeon(R) CPU E5-2680 v4
> > > CPU freq (hz):   33
> > > Cache line size: 64
> > > CPU count:   56
> > >
> > >
> > > CPU features supported:
> > > SSE3 PCLMULQDQ DTES64 MONITOR DS_CPL VMX SMX EIST TM2 SSSE3 FMA
> > CMPXCHG16B
> > > XTPR PDCM PCID DCA SSE4_1 SSE4_2 X2APIC MOVBE POPCNT TSC_DEADLINE
> > AES XSAVE
> > > OSXSAVE AVX F16C RDRAND FPU VME DE PSE TSC MSR PAE MCE CX8 APIC
> > SEP 

Re: [lng-odp] issues with usage of mellanox 100G NICs with ODP & ODP-DPDK

[Maxim Uvarov]

On 11/08/17 19:32, gyanesh patra wrote:
> I am not sure what you mean. Can you please elaborate?
> 
> As i mentioned before I am able to run dpdk examples. Hence the drivers
> are available and working fine.
> I configured ODP & ODP-DPDK with "LDFLAGS=-libverbs" and compiled to
> work with mellanox. I followed the same while compiling dpdk too.
> 
> Is there anything i am missing?
> 
> P Gyanesh Kumar Patra


in general if CONFIG_RTE_LIBRTE_MLX5_PMD=y was specified then it has to
work. I think we did test only with ixgbe. But in general it's common code.

"Unable to init any I/O type." means it it called all open for all pktio
in list here:
./platform/linux-generic/pktio/io_ops.c

and setup_pkt_dpdk() failed for some reason.

I do not like allocations errors in your log.

Try to compile ODP with --enable-debug-print --enable-debug it will make
ODP_DBG() macro work and it will be visible why it does not opens pktio.

Maxim


> 
> On Wed, Nov 8, 2017 at 5:22 PM, Maxim Uvarov  > wrote:
> 
> is Mellanox pmd compiled in?
> 
> Maxim.
> 
> On 11/08/17 17:58, gyanesh patra wrote:
> > Hi,
> > I am trying to run ODP & ODP-DPDK examples on our server with
> mellanox 100G
> > NICs. I am using the odp_l2fwd example. While running the example,
> I am
> > facing some issues.
> > -> When I run "ODP" example using the if names given by kernel as
> > arguments, I am not getting enough throughput.(the value is very low)
> > -> And when I try "ODP-DPDK" example using port ID as "0,1", it can't
> > create pktio. Whereas I am able to run the examples from "DPDK"
> > repo with portID "0,1" for the same mellanox NICs. I tried running
> with
> > "81:00.0,81:00.1" and also with if-names too without any success.
> Adding
> > the whitelist using ODP_PLATFORM_PARAMS doesn't help either.
> >
> > Am I missing any steps to use mellanox NICs? OR is there a
> different method
> > to specify the device details to create pktio?
> > I am providing the output of "odp_l2fwd" examples for ODP and ODP-DPDK
> > repository here.
> >
> > The NICs being used:
> >
> > :81:00.0 'MT27700 Family [ConnectX-4]' if=enp129s0f0 drv=mlx5_core
> > unused=
> > :81:00.1 'MT27700 Family [ConnectX-4]' if=enp129s0f1 drv=mlx5_core
> > unused=
> >
> > ODP l2fwd example run details:
> > --
> > root@ubuntu:/home/ubuntu/odp/test/performance# ./odp_l2fwd -i
> > enp129s0f0,enp129s0f1
> > HW time counter freq: 239886 
> <(239)%20999-9886> hz
> >
> > _ishmphy.c:152:_odp_ishmphy_map():mmap failed:Cannot allocate memory
> > _ishm.c:880:_odp_ishm_reserve():No huge pages, fall back to normal
> pages.
> > check: /proc/sys/vm/nr_hugepages.
> > _ishmphy.c:152:_odp_ishmphy_map():mmap failed:Cannot allocate memory
> > _ishmphy.c:152:_odp_ishmphy_map():mmap failed:Cannot allocate memory
> > _ishmphy.c:152:_odp_ishmphy_map():mmap failed:Cannot allocate memory
> >  PKTIO: initialized loop interface.
> >  PKTIO: initialized pcap interface.
> >  PKTIO: initialized ipc interface.
> >  PKTIO: initialized socket mmap, use export
> ODP_PKTIO_DISABLE_SOCKET_MMAP=1
> > to disable.
> >  PKTIO: initialized socket mmsg,use export
> ODP_PKTIO_DISABLE_SOCKET_MMSG=1
> > to disable.
> > _ishmphy.c:152:_odp_ishmphy_map():mmap failed:Cannot allocate memory
> > _ishmphy.c:152:_odp_ishmphy_map():mmap failed:Cannot allocate memory
> > _ishmphy.c:152:_odp_ishmphy_map():mmap failed:Cannot allocate memory
> > _ishmphy.c:152:_odp_ishmphy_map():mmap failed:Cannot allocate memory
> >
> > ODP system info
> > ---
> > ODP API version: 1.15.0
> > ODP impl name:   "odp-linux"
> > CPU model:       Intel(R) Xeon(R) CPU E5-2680 v4
> > CPU freq (hz):   33
> > Cache line size: 64
> > CPU count:       56
> >
> >
> > CPU features supported:
> > SSE3 PCLMULQDQ DTES64 MONITOR DS_CPL VMX SMX EIST TM2 SSSE3 FMA
> CMPXCHG16B
> > XTPR PDCM PCID DCA SSE4_1 SSE4_2 X2APIC MOVBE POPCNT TSC_DEADLINE
> AES XSAVE
> > OSXSAVE AVX F16C RDRAND FPU VME DE PSE TSC MSR PAE MCE CX8 APIC
> SEP MTRR
> > PGE MCA CMOV PAT PSE36 CLFSH DS ACPI MMX FXSR SSE SSE2 SS HTT TM PBE
> > DIGTEMP TRBOBST ARAT PLN ECMD PTM MPERF_APERF_MSR ENERGY_EFF
> FSGSBASE HLE
> > AVX2 BMI2 ERMS INVPCID RTM LAHF_SAHF SYSCALL XD 1GB_PG RDTSCP
> EM64T INVTSC
> >
> > CPU features NOT supported:
> > CNXT_ID PSN ACNT2 BMI1 SMEP AVX512F LZCNT
> >
> > Running ODP appl: "odp_l2fwd"
> > -
> > IF-count:        2
> > Using IFs:       enp129s0f0 enp129s0f1
> > Mode:            PKTIN_DIRECT, PKTOUT_DIRECT
> >
> > num worker threads: 32
> > first CPU:          24
> > cpu mask:      

Re: [lng-odp] issues with usage of mellanox 100G NICs with ODP & ODP-DPDK

[gyanesh patra]

I am not sure what you mean. Can you please elaborate?

As i mentioned before I am able to run dpdk examples. Hence the drivers are
available and working fine.
I configured ODP & ODP-DPDK with "LDFLAGS=-libverbs" and compiled to work
with mellanox. I followed the same while compiling dpdk too.

Is there anything i am missing?

P Gyanesh Kumar Patra

On Wed, Nov 8, 2017 at 5:22 PM, Maxim Uvarov 
wrote:

> is Mellanox pmd compiled in?
>
> Maxim.
>
> On 11/08/17 17:58, gyanesh patra wrote:
> > Hi,
> > I am trying to run ODP & ODP-DPDK examples on our server with mellanox
> 100G
> > NICs. I am using the odp_l2fwd example. While running the example, I am
> > facing some issues.
> > -> When I run "ODP" example using the if names given by kernel as
> > arguments, I am not getting enough throughput.(the value is very low)
> > -> And when I try "ODP-DPDK" example using port ID as "0,1", it can't
> > create pktio. Whereas I am able to run the examples from "DPDK"
> > repo with portID "0,1" for the same mellanox NICs. I tried running with
> > "81:00.0,81:00.1" and also with if-names too without any success. Adding
> > the whitelist using ODP_PLATFORM_PARAMS doesn't help either.
> >
> > Am I missing any steps to use mellanox NICs? OR is there a different
> method
> > to specify the device details to create pktio?
> > I am providing the output of "odp_l2fwd" examples for ODP and ODP-DPDK
> > repository here.
> >
> > The NICs being used:
> >
> > :81:00.0 'MT27700 Family [ConnectX-4]' if=enp129s0f0 drv=mlx5_core
> > unused=
> > :81:00.1 'MT27700 Family [ConnectX-4]' if=enp129s0f1 drv=mlx5_core
> > unused=
> >
> > ODP l2fwd example run details:
> > --
> > root@ubuntu:/home/ubuntu/odp/test/performance# ./odp_l2fwd -i
> > enp129s0f0,enp129s0f1
> > HW time counter freq: 239886 <(239)%20999-9886> hz
> >
> > _ishmphy.c:152:_odp_ishmphy_map():mmap failed:Cannot allocate memory
> > _ishm.c:880:_odp_ishm_reserve():No huge pages, fall back to normal
> pages.
> > check: /proc/sys/vm/nr_hugepages.
> > _ishmphy.c:152:_odp_ishmphy_map():mmap failed:Cannot allocate memory
> > _ishmphy.c:152:_odp_ishmphy_map():mmap failed:Cannot allocate memory
> > _ishmphy.c:152:_odp_ishmphy_map():mmap failed:Cannot allocate memory
> >  PKTIO: initialized loop interface.
> >  PKTIO: initialized pcap interface.
> >  PKTIO: initialized ipc interface.
> >  PKTIO: initialized socket mmap, use export
> ODP_PKTIO_DISABLE_SOCKET_MMAP=1
> > to disable.
> >  PKTIO: initialized socket mmsg,use export ODP_PKTIO_DISABLE_SOCKET_MMSG=
> 1
> > to disable.
> > _ishmphy.c:152:_odp_ishmphy_map():mmap failed:Cannot allocate memory
> > _ishmphy.c:152:_odp_ishmphy_map():mmap failed:Cannot allocate memory
> > _ishmphy.c:152:_odp_ishmphy_map():mmap failed:Cannot allocate memory
> > _ishmphy.c:152:_odp_ishmphy_map():mmap failed:Cannot allocate memory
> >
> > ODP system info
> > ---
> > ODP API version: 1.15.0
> > ODP impl name:   "odp-linux"
> > CPU model:   Intel(R) Xeon(R) CPU E5-2680 v4
> > CPU freq (hz):   33
> > Cache line size: 64
> > CPU count:   56
> >
> >
> > CPU features supported:
> > SSE3 PCLMULQDQ DTES64 MONITOR DS_CPL VMX SMX EIST TM2 SSSE3 FMA
> CMPXCHG16B
> > XTPR PDCM PCID DCA SSE4_1 SSE4_2 X2APIC MOVBE POPCNT TSC_DEADLINE AES
> XSAVE
> > OSXSAVE AVX F16C RDRAND FPU VME DE PSE TSC MSR PAE MCE CX8 APIC SEP MTRR
> > PGE MCA CMOV PAT PSE36 CLFSH DS ACPI MMX FXSR SSE SSE2 SS HTT TM PBE
> > DIGTEMP TRBOBST ARAT PLN ECMD PTM MPERF_APERF_MSR ENERGY_EFF FSGSBASE HLE
> > AVX2 BMI2 ERMS INVPCID RTM LAHF_SAHF SYSCALL XD 1GB_PG RDTSCP EM64T
> INVTSC
> >
> > CPU features NOT supported:
> > CNXT_ID PSN ACNT2 BMI1 SMEP AVX512F LZCNT
> >
> > Running ODP appl: "odp_l2fwd"
> > -
> > IF-count:2
> > Using IFs:   enp129s0f0 enp129s0f1
> > Mode:PKTIN_DIRECT, PKTOUT_DIRECT
> >
> > num worker threads: 32
> > first CPU:  24
> > cpu mask:   0x00
> >
> > _ishmphy.c:152:_odp_ishmphy_map():mmap failed:Cannot allocate memory
> > _ishmphy.c:152:_odp_ishmphy_map():mmap failed:Cannot allocate memory
> >
> > Pool info
> > -
> >   pool0
> >   namepacket pool
> >   pool type   packet
> >   pool shm11
> >   user area shm   0
> >   num 8192
> >   align   64
> >   headroom128
> >   seg len 8064
> >   max data len65536
> >   tailroom0
> >   block size  8768
> >   uarea size  0
> >   shm size72143104
> >   base addr   0x7f5fc1234000
> >   uarea shm size  0
> >   uarea base addr (nil)
> >
> > pktio/socket_mmap.c:401:mmap_setup_ring():setsockopt(pkt mmap): Invalid
> > argument
> > pktio/socket_mmap.c:496:sock_mmap_close():mmap_unmap_sock() Invalid
> argument
> > created pktio 1, dev: enp129s0f0, drv: socket
> > Sharing 1 input queues between 16 workers
> > Sharing 1 output queues between 16 workers
> > created 1 input and 1 output 

Re: [lng-odp] issues with usage of mellanox 100G NICs with ODP & ODP-DPDK

[Maxim Uvarov]

is Mellanox pmd compiled in?

Maxim.

On 11/08/17 17:58, gyanesh patra wrote:
> Hi,
> I am trying to run ODP & ODP-DPDK examples on our server with mellanox 100G
> NICs. I am using the odp_l2fwd example. While running the example, I am
> facing some issues.
> -> When I run "ODP" example using the if names given by kernel as
> arguments, I am not getting enough throughput.(the value is very low)
> -> And when I try "ODP-DPDK" example using port ID as "0,1", it can't
> create pktio. Whereas I am able to run the examples from "DPDK"
> repo with portID "0,1" for the same mellanox NICs. I tried running with
> "81:00.0,81:00.1" and also with if-names too without any success. Adding
> the whitelist using ODP_PLATFORM_PARAMS doesn't help either.
> 
> Am I missing any steps to use mellanox NICs? OR is there a different method
> to specify the device details to create pktio?
> I am providing the output of "odp_l2fwd" examples for ODP and ODP-DPDK
> repository here.
> 
> The NICs being used:
> 
> :81:00.0 'MT27700 Family [ConnectX-4]' if=enp129s0f0 drv=mlx5_core
> unused=
> :81:00.1 'MT27700 Family [ConnectX-4]' if=enp129s0f1 drv=mlx5_core
> unused=
> 
> ODP l2fwd example run details:
> --
> root@ubuntu:/home/ubuntu/odp/test/performance# ./odp_l2fwd -i
> enp129s0f0,enp129s0f1
> HW time counter freq: 239886 <(239)%20999-9886> hz
> 
> _ishmphy.c:152:_odp_ishmphy_map():mmap failed:Cannot allocate memory
> _ishm.c:880:_odp_ishm_reserve():No huge pages, fall back to normal pages.
> check: /proc/sys/vm/nr_hugepages.
> _ishmphy.c:152:_odp_ishmphy_map():mmap failed:Cannot allocate memory
> _ishmphy.c:152:_odp_ishmphy_map():mmap failed:Cannot allocate memory
> _ishmphy.c:152:_odp_ishmphy_map():mmap failed:Cannot allocate memory
>  PKTIO: initialized loop interface.
>  PKTIO: initialized pcap interface.
>  PKTIO: initialized ipc interface.
>  PKTIO: initialized socket mmap, use export ODP_PKTIO_DISABLE_SOCKET_MMAP=1
> to disable.
>  PKTIO: initialized socket mmsg,use export ODP_PKTIO_DISABLE_SOCKET_MMSG=1
> to disable.
> _ishmphy.c:152:_odp_ishmphy_map():mmap failed:Cannot allocate memory
> _ishmphy.c:152:_odp_ishmphy_map():mmap failed:Cannot allocate memory
> _ishmphy.c:152:_odp_ishmphy_map():mmap failed:Cannot allocate memory
> _ishmphy.c:152:_odp_ishmphy_map():mmap failed:Cannot allocate memory
> 
> ODP system info
> ---
> ODP API version: 1.15.0
> ODP impl name:   "odp-linux"
> CPU model:   Intel(R) Xeon(R) CPU E5-2680 v4
> CPU freq (hz):   33
> Cache line size: 64
> CPU count:   56
> 
> 
> CPU features supported:
> SSE3 PCLMULQDQ DTES64 MONITOR DS_CPL VMX SMX EIST TM2 SSSE3 FMA CMPXCHG16B
> XTPR PDCM PCID DCA SSE4_1 SSE4_2 X2APIC MOVBE POPCNT TSC_DEADLINE AES XSAVE
> OSXSAVE AVX F16C RDRAND FPU VME DE PSE TSC MSR PAE MCE CX8 APIC SEP MTRR
> PGE MCA CMOV PAT PSE36 CLFSH DS ACPI MMX FXSR SSE SSE2 SS HTT TM PBE
> DIGTEMP TRBOBST ARAT PLN ECMD PTM MPERF_APERF_MSR ENERGY_EFF FSGSBASE HLE
> AVX2 BMI2 ERMS INVPCID RTM LAHF_SAHF SYSCALL XD 1GB_PG RDTSCP EM64T INVTSC
> 
> CPU features NOT supported:
> CNXT_ID PSN ACNT2 BMI1 SMEP AVX512F LZCNT
> 
> Running ODP appl: "odp_l2fwd"
> -
> IF-count:2
> Using IFs:   enp129s0f0 enp129s0f1
> Mode:PKTIN_DIRECT, PKTOUT_DIRECT
> 
> num worker threads: 32
> first CPU:  24
> cpu mask:   0x00
> 
> _ishmphy.c:152:_odp_ishmphy_map():mmap failed:Cannot allocate memory
> _ishmphy.c:152:_odp_ishmphy_map():mmap failed:Cannot allocate memory
> 
> Pool info
> -
>   pool0
>   namepacket pool
>   pool type   packet
>   pool shm11
>   user area shm   0
>   num 8192
>   align   64
>   headroom128
>   seg len 8064
>   max data len65536
>   tailroom0
>   block size  8768
>   uarea size  0
>   shm size72143104
>   base addr   0x7f5fc1234000
>   uarea shm size  0
>   uarea base addr (nil)
> 
> pktio/socket_mmap.c:401:mmap_setup_ring():setsockopt(pkt mmap): Invalid
> argument
> pktio/socket_mmap.c:496:sock_mmap_close():mmap_unmap_sock() Invalid argument
> created pktio 1, dev: enp129s0f0, drv: socket
> Sharing 1 input queues between 16 workers
> Sharing 1 output queues between 16 workers
> created 1 input and 1 output queues on (enp129s0f0)
> pktio/socket_mmap.c:401:mmap_setup_ring():setsockopt(pkt mmap): Invalid
> argument
> pktio/socket_mmap.c:496:sock_mmap_close():mmap_unmap_sock() Invalid argument
> created pktio 2, dev: enp129s0f1, drv: socket
> Sharing 1 input queues between 16 workers
> Sharing 1 output queues between 16 workers
> created 1 input and 1 output queues on (enp129s0f1)
> 
> Queue binding (indexes)
> ---
> worker 0
>   rx: pktio 0, queue 0
>   tx: pktio 1, queue 0
> worker 1
>   rx: pktio 1, queue 0
>   tx: pktio 0, queue 0
> worker 2
>   rx: pktio 0, queue 0
>   tx: pktio 1, queue 0
> worker 3
>   rx: pktio 1, queue 0
>   tx: